Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 20:08:23.645624 14763 test_util.cc:268] Using random seed: 209409423
I20240501 20:08:23.656534 14763 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:23.656610 14763 ts_itest-base.cc:113] --------------
I20240501 20:08:23.656651 14763 ts_itest-base.cc:114] 8 tablet servers
I20240501 20:08:23.656694 14763 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:23.656746 14763 ts_itest-base.cc:116] --------------
2024-05-01T20:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:23Z Disabled control of system clock
I20240501 20:08:23.680521 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:34075
--webserver_interface=127.14.106.254
--webserver_port=0
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.106.254:34075
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:08:23.994103 14775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:23.994396 14775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:23.994488 14775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:24.001652 14775 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:24.001799 14775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:24.001857 14775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:24.001914 14775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:24.001965 14775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:24.010607 14775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.106.254:34075
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:34075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:24.011807 14775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:24.013429 14775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:24.024513 14780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:24.025317 14781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:24.025995 14783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:24.026916 14775 server_base.cc:1034] running on GCE node
I20240501 20:08:24.027669 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:24.029052 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:24.030239 14775 hybrid_clock.cc:648] HybridClock initialized: now 1714594104030206 us; error 36 us; skew 500 ppm
I20240501 20:08:24.030566 14775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:24.037444 14775 webserver.cc:460] Webserver started at http://127.14.106.254:33391/ using document root <none> and password file <none>
I20240501 20:08:24.038173 14775 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:24.038295 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:24.038586 14775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:24.041123 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "de58c25b144942ea83d7b9df7f3e4c21"
format_stamp: "Formatted at 2024-05-01 20:08:24 on dist-test-slave-pg4x"
I20240501 20:08:24.041895 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "de58c25b144942ea83d7b9df7f3e4c21"
format_stamp: "Formatted at 2024-05-01 20:08:24 on dist-test-slave-pg4x"
I20240501 20:08:24.046854 14775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 20:08:24.050052 14789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:24.051364 14775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:24.051519 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
uuid: "de58c25b144942ea83d7b9df7f3e4c21"
format_stamp: "Formatted at 2024-05-01 20:08:24 on dist-test-slave-pg4x"
I20240501 20:08:24.051694 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:24.083478 14775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:24.084276 14775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:24.084591 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:24.102046 14775 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.254:34075
I20240501 20:08:24.102068 14840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.254:34075 every 8 connection(s)
I20240501 20:08:24.103384 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:24.105562 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 14775
I20240501 20:08:24.105796 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:24.107117 14841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:24.117336 14841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21: Bootstrap starting.
I20240501 20:08:24.120348 14841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:24.121337 14841 log.cc:824] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:24.123792 14841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21: No bootstrap required, opened a new log
I20240501 20:08:24.130118 14841 raft_consensus.cc:348] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de58c25b144942ea83d7b9df7f3e4c21" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34075 } }
I20240501 20:08:24.130457 14841 raft_consensus.cc:374] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:24.130559 14841 raft_consensus.cc:729] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de58c25b144942ea83d7b9df7f3e4c21, State: Initialized, Role: FOLLOWER
I20240501 20:08:24.131198 14841 consensus_queue.cc:260] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de58c25b144942ea83d7b9df7f3e4c21" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34075 } }
I20240501 20:08:24.131438 14841 raft_consensus.cc:388] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:24.131582 14841 raft_consensus.cc:482] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:24.131844 14841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:24.133924 14841 raft_consensus.cc:504] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de58c25b144942ea83d7b9df7f3e4c21" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34075 } }
I20240501 20:08:24.134373 14841 leader_election.cc:302] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de58c25b144942ea83d7b9df7f3e4c21; no voters:
I20240501 20:08:24.134986 14841 leader_election.cc:288] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:24.135090 14846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:24.135725 14846 raft_consensus.cc:686] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [term 1 LEADER]: Becoming Leader. State: Replica: de58c25b144942ea83d7b9df7f3e4c21, State: Running, Role: LEADER
I20240501 20:08:24.136221 14846 consensus_queue.cc:237] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de58c25b144942ea83d7b9df7f3e4c21" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34075 } }
I20240501 20:08:24.136829 14841 sys_catalog.cc:564] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:24.139032 14848 sys_catalog.cc:455] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de58c25b144942ea83d7b9df7f3e4c21. Latest consensus state: current_term: 1 leader_uuid: "de58c25b144942ea83d7b9df7f3e4c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de58c25b144942ea83d7b9df7f3e4c21" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34075 } } }
I20240501 20:08:24.139452 14848 sys_catalog.cc:458] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:24.139909 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de58c25b144942ea83d7b9df7f3e4c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de58c25b144942ea83d7b9df7f3e4c21" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34075 } } }
I20240501 20:08:24.142867 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:24.143821 14854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:24.148948 14854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:24.156774 14854 catalog_manager.cc:1347] Generated new cluster ID: dbeebf84ecb04c34a5bc64edf9620131
I20240501 20:08:24.156884 14854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:24.175607 14854 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:24.177232 14854 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:24.192792 14854 catalog_manager.cc:5857] T 00000000000000000000000000000000 P de58c25b144942ea83d7b9df7f3e4c21: Generated new TSK 0
I20240501 20:08:24.193836 14854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:24.206984 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.193:0
--local_ip_for_outbound_sockets=127.14.106.193
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34075
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 20:08:24.528802 14865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:24.529101 14865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:24.529261 14865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:24.536697 14865 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:24.536831 14865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:24.536989 14865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.193
I20240501 20:08:24.546069 14865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.106.193
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.193
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:24.547420 14865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:24.549134 14865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:24.562307 14870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:24.563176 14871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:24.563820 14873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:24.565065 14865 server_base.cc:1034] running on GCE node
I20240501 20:08:24.565683 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:24.566727 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:24.567968 14865 hybrid_clock.cc:648] HybridClock initialized: now 1714594104567914 us; error 40 us; skew 500 ppm
I20240501 20:08:24.568322 14865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:24.570713 14865 webserver.cc:460] Webserver started at http://127.14.106.193:40831/ using document root <none> and password file <none>
I20240501 20:08:24.571470 14865 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:24.571575 14865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:24.571897 14865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:24.574569 14865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7decc3355b114404abddb7a10cc4358a"
format_stamp: "Formatted at 2024-05-01 20:08:24 on dist-test-slave-pg4x"
I20240501 20:08:24.575383 14865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7decc3355b114404abddb7a10cc4358a"
format_stamp: "Formatted at 2024-05-01 20:08:24 on dist-test-slave-pg4x"
I20240501 20:08:24.580679 14865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:24.584287 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:24.585657 14865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:24.585822 14865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7decc3355b114404abddb7a10cc4358a"
format_stamp: "Formatted at 2024-05-01 20:08:24 on dist-test-slave-pg4x"
I20240501 20:08:24.586024 14865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:24.609750 14865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:24.610667 14865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:24.610946 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:24.611900 14865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:24.613505 14865 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:24.613603 14865 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:24.613785 14865 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:24.613915 14865 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:24.646124 14865 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.193:44871
I20240501 20:08:24.646176 14991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.193:44871 every 8 connection(s)
I20240501 20:08:24.647442 14865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:24.654476 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 14865
I20240501 20:08:24.654712 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:24.657931 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.194:0
--local_ip_for_outbound_sockets=127.14.106.194
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34075
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:24.660375 14992 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:24.660672 14992 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:24.661448 14992 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:24.663393 14806 ts_manager.cc:194] Registered new tserver with Master: 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:24.664795 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.193:59775
W20240501 20:08:24.979147 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:24.979431 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:24.979589 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:24.986907 14996 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:24.987015 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:24.987186 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.194
I20240501 20:08:24.995867 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.106.194
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.194
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:24.997090 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:24.998721 14996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:25.010818 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:25.011550 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:25.012390 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:25.013221 14996 server_base.cc:1034] running on GCE node
I20240501 20:08:25.013784 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:25.014782 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:25.016021 14996 hybrid_clock.cc:648] HybridClock initialized: now 1714594105015970 us; error 39 us; skew 500 ppm
I20240501 20:08:25.016325 14996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:25.018514 14996 webserver.cc:460] Webserver started at http://127.14.106.194:43033/ using document root <none> and password file <none>
I20240501 20:08:25.019270 14996 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:25.019390 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:25.019701 14996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:25.022310 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.023042 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.027851 14996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240501 20:08:25.031025 15010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.032241 14996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:25.032394 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.032577 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:25.050978 14996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:25.051712 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:25.051965 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:25.052846 14996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:25.054211 14996 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:25.054312 14996 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.054428 14996 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:25.054504 14996 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.085620 14996 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.194:34059
I20240501 20:08:25.085678 15122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.194:34059 every 8 connection(s)
I20240501 20:08:25.086891 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:25.094820 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 14996
I20240501 20:08:25.094967 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:25.097888 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.195:0
--local_ip_for_outbound_sockets=127.14.106.195
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34075
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:25.099141 15123 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:25.099421 15123 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:25.100173 15123 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:25.101603 14806 ts_manager.cc:194] Registered new tserver with Master: 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:25.102370 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.194:50929
W20240501 20:08:25.414260 15126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:25.414536 15126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:25.414709 15126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:25.422083 15126 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:25.422214 15126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:25.422387 15126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.195
I20240501 20:08:25.431267 15126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.106.195
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.195
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:25.432538 15126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:25.434206 15126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:25.446520 15131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:25.447242 15134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:25.448065 15132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:25.448913 15126 server_base.cc:1034] running on GCE node
I20240501 20:08:25.449470 15126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:25.450369 15126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:25.451570 15126 hybrid_clock.cc:648] HybridClock initialized: now 1714594105451521 us; error 37 us; skew 500 ppm
I20240501 20:08:25.451900 15126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:25.454123 15126 webserver.cc:460] Webserver started at http://127.14.106.195:41197/ using document root <none> and password file <none>
I20240501 20:08:25.454860 15126 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:25.454975 15126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:25.455369 15126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:25.458125 15126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2177fb70b1324acb81b2bea7be19955d"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.458851 15126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2177fb70b1324acb81b2bea7be19955d"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.463618 15126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 20:08:25.466792 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.467975 15126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:08:25.468116 15126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2177fb70b1324acb81b2bea7be19955d"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.468299 15126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:25.491348 15126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:25.492089 15126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:25.492341 15126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:25.493207 15126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:25.494968 15126 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:25.495066 15126 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.495177 15126 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:25.495249 15126 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.525988 15126 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.195:34095
I20240501 20:08:25.526036 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.195:34095 every 8 connection(s)
I20240501 20:08:25.527223 15126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:25.534797 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15126
I20240501 20:08:25.534992 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:25.538192 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:0
--local_ip_for_outbound_sockets=127.14.106.196
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34075
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:25.539435 15253 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:25.539664 15253 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:25.540347 15253 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:25.541570 14806 ts_manager.cc:194] Registered new tserver with Master: 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095)
I20240501 20:08:25.542711 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.195:54849
I20240501 20:08:25.667383 14992 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
W20240501 20:08:25.855044 15256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:25.855314 15256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:25.855448 15256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:25.862784 15256 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:25.862895 15256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:25.863070 15256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:08:25.871871 15256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:25.873085 15256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:25.874756 15256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:25.886499 15261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:25.887334 15262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:25.888110 15264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:25.889017 15256 server_base.cc:1034] running on GCE node
I20240501 20:08:25.889623 15256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:25.890534 15256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:25.891763 15256 hybrid_clock.cc:648] HybridClock initialized: now 1714594105891706 us; error 48 us; skew 500 ppm
I20240501 20:08:25.892081 15256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:25.894276 15256 webserver.cc:460] Webserver started at http://127.14.106.196:38771/ using document root <none> and password file <none>
I20240501 20:08:25.895009 15256 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:25.895129 15256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:25.895444 15256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:25.898047 15256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e494ca2918594ba792c2d37d98b8c607"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.898833 15256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e494ca2918594ba792c2d37d98b8c607"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.903800 15256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:25.906989 15270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.908365 15256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:08:25.908530 15256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e494ca2918594ba792c2d37d98b8c607"
format_stamp: "Formatted at 2024-05-01 20:08:25 on dist-test-slave-pg4x"
I20240501 20:08:25.908721 15256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:25.938045 15256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:25.938935 15256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:25.939198 15256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:25.940130 15256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:25.941569 15256 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:25.941668 15256 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.941803 15256 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:25.941917 15256 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.973780 15256 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:39333
I20240501 20:08:25.973843 15382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:39333 every 8 connection(s)
I20240501 20:08:25.975131 15256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:25.984730 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15256
I20240501 20:08:25.984925 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:25.987121 15383 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:25.987430 15383 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:25.988143 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:0
--local_ip_for_outbound_sockets=127.14.106.197
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34075
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:25.988173 15383 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:25.989490 14806 ts_manager.cc:194] Registered new tserver with Master: e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:25.990270 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:40173
I20240501 20:08:26.105155 15123 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
W20240501 20:08:26.309336 15386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:26.309626 15386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:26.309784 15386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:26.317226 15386 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:26.317356 15386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:26.317513 15386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:08:26.326404 15386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:26.327692 15386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:26.329365 15386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:26.341889 15391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:26.342483 15394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:26.343350 15392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:26.344195 15386 server_base.cc:1034] running on GCE node
I20240501 20:08:26.344782 15386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:26.345690 15386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:26.346901 15386 hybrid_clock.cc:648] HybridClock initialized: now 1714594106346857 us; error 39 us; skew 500 ppm
I20240501 20:08:26.347210 15386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:26.349591 15386 webserver.cc:460] Webserver started at http://127.14.106.197:33771/ using document root <none> and password file <none>
I20240501 20:08:26.350292 15386 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:26.350409 15386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:26.350772 15386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:26.353371 15386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "250badc514c94bcdbe2ace1b62a0874f"
format_stamp: "Formatted at 2024-05-01 20:08:26 on dist-test-slave-pg4x"
I20240501 20:08:26.354094 15386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "250badc514c94bcdbe2ace1b62a0874f"
format_stamp: "Formatted at 2024-05-01 20:08:26 on dist-test-slave-pg4x"
I20240501 20:08:26.359213 15386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20240501 20:08:26.362427 15400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.363735 15386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:26.363905 15386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "250badc514c94bcdbe2ace1b62a0874f"
format_stamp: "Formatted at 2024-05-01 20:08:26 on dist-test-slave-pg4x"
I20240501 20:08:26.364094 15386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:26.378798 15386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:26.379531 15386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:26.379802 15386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:26.380652 15386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:26.382035 15386 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:26.382130 15386 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.382246 15386 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:26.382321 15386 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.414700 15386 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:35831
I20240501 20:08:26.414783 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:35831 every 8 connection(s)
I20240501 20:08:26.415993 15386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:26.424122 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15386
I20240501 20:08:26.424316 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:26.427373 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.198:0
--local_ip_for_outbound_sockets=127.14.106.198
--webserver_interface=127.14.106.198
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34075
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:26.428375 15513 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:26.428640 15513 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:26.429400 15513 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:26.430770 14806 ts_manager.cc:194] Registered new tserver with Master: 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197:35831)
I20240501 20:08:26.431545 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:60119
I20240501 20:08:26.545506 15253 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
W20240501 20:08:26.747669 15516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:26.747975 15516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:26.748129 15516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:26.755539 15516 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:26.755674 15516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:26.755857 15516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.198
I20240501 20:08:26.764792 15516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.106.198
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.198
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:26.766054 15516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:26.767725 15516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:26.779613 15522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:26.780381 15521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:26.781148 15524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:26.782022 15516 server_base.cc:1034] running on GCE node
I20240501 20:08:26.782536 15516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:26.783442 15516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:26.784639 15516 hybrid_clock.cc:648] HybridClock initialized: now 1714594106784598 us; error 39 us; skew 500 ppm
I20240501 20:08:26.784941 15516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:26.787089 15516 webserver.cc:460] Webserver started at http://127.14.106.198:33963/ using document root <none> and password file <none>
I20240501 20:08:26.787762 15516 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:26.787891 15516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:26.788197 15516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:26.790772 15516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "98d6b3fd85674334a24cd273bf525324"
format_stamp: "Formatted at 2024-05-01 20:08:26 on dist-test-slave-pg4x"
I20240501 20:08:26.791541 15516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "98d6b3fd85674334a24cd273bf525324"
format_stamp: "Formatted at 2024-05-01 20:08:26 on dist-test-slave-pg4x"
I20240501 20:08:26.796372 15516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:08:26.799425 15530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.800586 15516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:08:26.800733 15516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "98d6b3fd85674334a24cd273bf525324"
format_stamp: "Formatted at 2024-05-01 20:08:26 on dist-test-slave-pg4x"
I20240501 20:08:26.800925 15516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:26.839550 15516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:26.840384 15516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:26.840646 15516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:26.841594 15516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:26.843075 15516 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:26.843173 15516 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.843307 15516 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:26.843389 15516 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.874708 15516 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.198:38031
I20240501 20:08:26.874773 15642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.198:38031 every 8 connection(s)
I20240501 20:08:26.876036 15516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:08:26.884438 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15516
I20240501 20:08:26.884626 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:08:26.888132 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.199:0
--local_ip_for_outbound_sockets=127.14.106.199
--webserver_interface=127.14.106.199
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34075
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:26.888551 15643 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:26.888828 15643 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:26.889642 15643 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:26.891032 14806 ts_manager.cc:194] Registered new tserver with Master: 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031)
I20240501 20:08:26.891976 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.198:40115
I20240501 20:08:26.993044 15383 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
W20240501 20:08:27.223642 15646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:27.223944 15646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:27.224100 15646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:27.231986 15646 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:27.232122 15646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:27.232288 15646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.199
I20240501 20:08:27.241298 15646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.106.199
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.199
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:27.243137 15646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:27.244940 15646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:27.257277 15651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:27.258033 15652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:27.258792 15654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:27.259589 15646 server_base.cc:1034] running on GCE node
I20240501 20:08:27.260207 15646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:27.261119 15646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:27.262317 15646 hybrid_clock.cc:648] HybridClock initialized: now 1714594107262267 us; error 48 us; skew 500 ppm
I20240501 20:08:27.262648 15646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:27.265285 15646 webserver.cc:460] Webserver started at http://127.14.106.199:45985/ using document root <none> and password file <none>
I20240501 20:08:27.265980 15646 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:27.266093 15646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:27.266395 15646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:27.269140 15646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "8dddd9d9b8fb47cb944ad4179152a915"
format_stamp: "Formatted at 2024-05-01 20:08:27 on dist-test-slave-pg4x"
I20240501 20:08:27.269917 15646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "8dddd9d9b8fb47cb944ad4179152a915"
format_stamp: "Formatted at 2024-05-01 20:08:27 on dist-test-slave-pg4x"
I20240501 20:08:27.275029 15646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20240501 20:08:27.278203 15660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.279606 15646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:27.279771 15646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "8dddd9d9b8fb47cb944ad4179152a915"
format_stamp: "Formatted at 2024-05-01 20:08:27 on dist-test-slave-pg4x"
I20240501 20:08:27.279973 15646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:27.315519 15646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:27.316378 15646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:27.316644 15646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:27.317590 15646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:27.319103 15646 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:27.319710 15646 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.319834 15646 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:27.319908 15646 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.358665 15646 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.199:38431
I20240501 20:08:27.358734 15772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.199:38431 every 8 connection(s)
I20240501 20:08:27.360004 15646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:08:27.366091 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15646
I20240501 20:08:27.366299 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:08:27.369719 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.200:0
--local_ip_for_outbound_sockets=127.14.106.200
--webserver_interface=127.14.106.200
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34075
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:27.373764 15773 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:27.374123 15773 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:27.375234 15773 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:27.376823 14806 ts_manager.cc:194] Registered new tserver with Master: 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199:38431)
I20240501 20:08:27.377998 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.199:57121
I20240501 20:08:27.433845 15513 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
W20240501 20:08:27.713454 15776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:27.713749 15776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:27.713894 15776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:27.721283 15776 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:27.721400 15776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:27.721567 15776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.200
I20240501 20:08:27.731300 15776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.106.200
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.200
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:27.732645 15776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:27.734416 15776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:27.748462 15781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:27.749145 15782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:27.750002 15784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:27.750939 15776 server_base.cc:1034] running on GCE node
I20240501 20:08:27.751670 15776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:27.752784 15776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:27.754000 15776 hybrid_clock.cc:648] HybridClock initialized: now 1714594107753944 us; error 45 us; skew 500 ppm
I20240501 20:08:27.754328 15776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:27.756870 15776 webserver.cc:460] Webserver started at http://127.14.106.200:41031/ using document root <none> and password file <none>
I20240501 20:08:27.757645 15776 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:27.757762 15776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:27.758083 15776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:27.760756 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0d9d84f0f3224d1a8fbaedd1021d730f"
format_stamp: "Formatted at 2024-05-01 20:08:27 on dist-test-slave-pg4x"
I20240501 20:08:27.761538 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0d9d84f0f3224d1a8fbaedd1021d730f"
format_stamp: "Formatted at 2024-05-01 20:08:27 on dist-test-slave-pg4x"
I20240501 20:08:27.767577 15776 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20240501 20:08:27.771179 15790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.772557 15776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:27.772717 15776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0d9d84f0f3224d1a8fbaedd1021d730f"
format_stamp: "Formatted at 2024-05-01 20:08:27 on dist-test-slave-pg4x"
I20240501 20:08:27.772930 15776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:27.795684 15776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:27.796573 15776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:27.796856 15776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:27.797859 15776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:27.799372 15776 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:27.799479 15776 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.799594 15776 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:27.799665 15776 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.835253 15776 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.200:40973
I20240501 20:08:27.835299 15902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.200:40973 every 8 connection(s)
I20240501 20:08:27.836536 15776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:08:27.838685 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15776
I20240501 20:08:27.838887 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:08:27.849637 15903 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:27.849901 15903 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:27.850697 15903 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:27.852020 14806 ts_manager.cc:194] Registered new tserver with Master: 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973)
I20240501 20:08:27.852808 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.200:42061
I20240501 20:08:27.856439 14763 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:08:27.877285 14763 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:08:27.878036 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15386
I20240501 20:08:27.886672 15508 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:27.894835 15643 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
I20240501 20:08:27.968309 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15386
I20240501 20:08:27.984812 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15516
I20240501 20:08:27.992831 15638 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:28.074213 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15516
I20240501 20:08:28.096729 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15646
I20240501 20:08:28.105049 15768 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:28.185681 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15646
I20240501 20:08:28.202332 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15776
I20240501 20:08:28.210834 15898 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:28.292013 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15776
I20240501 20:08:32.061102 14806 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60302:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:32.095703 15188 tablet_service.cc:1467] Processing CreateTablet for tablet e093bed4aae0484faa3cba7a4cc52ff9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.097177 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e093bed4aae0484faa3cba7a4cc52ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.102116 15317 tablet_service.cc:1467] Processing CreateTablet for tablet 69d637373a3b4d2a9d807ccdb716569e (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.102142 15057 tablet_service.cc:1467] Processing CreateTablet for tablet dcc9afae97ec4d6fabefad1a319e913c (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.102314 15056 tablet_service.cc:1467] Processing CreateTablet for tablet 69d637373a3b4d2a9d807ccdb716569e (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.102182 15058 tablet_service.cc:1467] Processing CreateTablet for tablet e093bed4aae0484faa3cba7a4cc52ff9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.103610 15057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc9afae97ec4d6fabefad1a319e913c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.103780 15317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d637373a3b4d2a9d807ccdb716569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.104593 15187 tablet_service.cc:1467] Processing CreateTablet for tablet dcc9afae97ec4d6fabefad1a319e913c (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.105015 15187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc9afae97ec4d6fabefad1a319e913c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.102949 15318 tablet_service.cc:1467] Processing CreateTablet for tablet e093bed4aae0484faa3cba7a4cc52ff9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.107017 15318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e093bed4aae0484faa3cba7a4cc52ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.110370 14927 tablet_service.cc:1467] Processing CreateTablet for tablet dcc9afae97ec4d6fabefad1a319e913c (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.111490 14926 tablet_service.cc:1467] Processing CreateTablet for tablet 69d637373a3b4d2a9d807ccdb716569e (DEFAULT_TABLE table=rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:32.111682 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc9afae97ec4d6fabefad1a319e913c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.112116 14926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d637373a3b4d2a9d807ccdb716569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.112464 15056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d637373a3b4d2a9d807ccdb716569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.114953 15058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e093bed4aae0484faa3cba7a4cc52ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.138020 15933 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:32.143534 15934 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:32.143585 15933 tablet_bootstrap.cc:654] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.145035 15933 log.cc:824] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:32.148380 15934 tablet_bootstrap.cc:654] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.148905 15935 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:32.149926 15934 log.cc:824] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:32.150360 15932 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:32.153771 15935 tablet_bootstrap.cc:654] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.154737 15932 tablet_bootstrap.cc:654] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.155038 15933 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:32.155112 15935 log.cc:824] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:32.155441 15933 ts_tablet_manager.cc:1367] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20240501 20:08:32.156100 15932 log.cc:824] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:32.163900 15932 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:32.164029 15935 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:32.164237 15932 ts_tablet_manager.cc:1367] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240501 20:08:32.164350 15934 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:32.164352 15935 ts_tablet_manager.cc:1367] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240501 20:08:32.164651 15934 ts_tablet_manager.cc:1367] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.012s
I20240501 20:08:32.166568 15933 raft_consensus.cc:348] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.167176 15933 raft_consensus.cc:374] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.167323 15933 raft_consensus.cc:729] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.168176 15933 consensus_queue.cc:260] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.173449 15935 raft_consensus.cc:348] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.173928 15935 raft_consensus.cc:374] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.174086 15935 raft_consensus.cc:729] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.174338 15934 raft_consensus.cc:348] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.174978 15934 raft_consensus.cc:374] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.174922 15935 consensus_queue.cc:260] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.175110 15934 raft_consensus.cc:729] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.175526 15932 raft_consensus.cc:348] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.175974 15932 raft_consensus.cc:374] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.176107 15932 raft_consensus.cc:729] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.176028 15934 consensus_queue.cc:260] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.176590 15933 ts_tablet_manager.cc:1398] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.021s user 0.011s sys 0.007s
I20240501 20:08:32.176896 15932 consensus_queue.cc:260] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.183357 15935 ts_tablet_manager.cc:1398] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20240501 20:08:32.183813 15935 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:32.184312 15934 ts_tablet_manager.cc:1398] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.020s user 0.003s sys 0.013s
I20240501 20:08:32.184789 15934 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:32.185104 15933 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:32.185599 15935 tablet_bootstrap.cc:654] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.186409 15934 tablet_bootstrap.cc:654] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.186857 15933 tablet_bootstrap.cc:654] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.191829 15932 ts_tablet_manager.cc:1398] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.027s user 0.018s sys 0.000s
I20240501 20:08:32.192416 15932 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:32.193724 15935 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:32.193894 15933 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:32.193924 15935 ts_tablet_manager.cc:1367] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240501 20:08:32.194191 15933 ts_tablet_manager.cc:1367] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20240501 20:08:32.194339 15932 tablet_bootstrap.cc:654] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.195097 15935 raft_consensus.cc:348] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.195274 15933 raft_consensus.cc:348] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.195477 15935 raft_consensus.cc:374] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.195595 15935 raft_consensus.cc:729] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.195777 15933 raft_consensus.cc:374] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.195943 15933 raft_consensus.cc:729] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.195896 15935 consensus_queue.cc:260] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.196285 15933 consensus_queue.cc:260] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.196725 15935 ts_tablet_manager.cc:1398] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:32.197813 15933 ts_tablet_manager.cc:1398] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:32.199092 15934 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:32.199218 15932 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:32.199410 15932 ts_tablet_manager.cc:1367] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:08:32.199296 15934 ts_tablet_manager.cc:1367] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240501 20:08:32.200462 15932 raft_consensus.cc:348] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.200709 15932 raft_consensus.cc:374] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.200490 15934 raft_consensus.cc:348] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.200834 15934 raft_consensus.cc:374] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.200881 15932 raft_consensus.cc:729] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.201117 15934 raft_consensus.cc:729] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.201122 15932 consensus_queue.cc:260] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.201347 15934 consensus_queue.cc:260] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.201725 15932 ts_tablet_manager.cc:1398] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:32.201864 15934 ts_tablet_manager.cc:1398] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:08:32.202019 15932 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:32.204435 15932 tablet_bootstrap.cc:654] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:32.206271 15932 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:32.206414 15932 ts_tablet_manager.cc:1367] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:08:32.206982 15932 raft_consensus.cc:348] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.207155 15932 raft_consensus.cc:374] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:32.207237 15932 raft_consensus.cc:729] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.207432 15932 consensus_queue.cc:260] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.207854 15932 ts_tablet_manager.cc:1398] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240501 20:08:32.230923 15384 tablet.cc:2369] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:32.241257 15943 raft_consensus.cc:482] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:32.241469 15943 raft_consensus.cc:504] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.243167 15943 leader_election.cc:288] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:32.250995 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d637373a3b4d2a9d807ccdb716569e" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:32.251021 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d637373a3b4d2a9d807ccdb716569e" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a" is_pre_election: true
I20240501 20:08:32.251422 14947 raft_consensus.cc:2454] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:32.251430 15338 raft_consensus.cc:2454] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:32.251950 15011 leader_election.cc:302] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:32.252298 15943 raft_consensus.cc:2781] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:32.252449 15943 raft_consensus.cc:482] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:32.252576 15943 raft_consensus.cc:3037] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.254655 15943 raft_consensus.cc:504] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.255220 15943 leader_election.cc:288] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:32.255615 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d637373a3b4d2a9d807ccdb716569e" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a"
I20240501 20:08:32.255702 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d637373a3b4d2a9d807ccdb716569e" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:32.255820 14947 raft_consensus.cc:3037] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.255906 15338 raft_consensus.cc:3037] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.258900 14947 raft_consensus.cc:2454] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:32.258900 15338 raft_consensus.cc:2454] T 69d637373a3b4d2a9d807ccdb716569e P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:32.259303 15011 leader_election.cc:302] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:32.259572 15943 raft_consensus.cc:2781] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:32.259999 15943 raft_consensus.cc:686] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Running, Role: LEADER
I20240501 20:08:32.260493 15943 consensus_queue.cc:237] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.264789 14805 catalog_manager.cc:5484] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } }
W20240501 20:08:32.283180 15254 tablet.cc:2369] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:32.309263 15942 raft_consensus.cc:482] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:32.309468 15942 raft_consensus.cc:504] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.311054 15942 leader_election.cc:288] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:32.318476 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc9afae97ec4d6fabefad1a319e913c" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a" is_pre_election: true
I20240501 20:08:32.318755 14947 raft_consensus.cc:2454] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 0.
I20240501 20:08:32.319192 15141 leader_election.cc:302] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:32.319342 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc9afae97ec4d6fabefad1a319e913c" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" is_pre_election: true
I20240501 20:08:32.319553 15942 raft_consensus.cc:2781] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:32.319691 15942 raft_consensus.cc:482] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:32.319732 15078 raft_consensus.cc:2454] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 0.
I20240501 20:08:32.319801 15942 raft_consensus.cc:3037] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.321950 15942 raft_consensus.cc:504] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.322538 15942 leader_election.cc:288] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 election: Requested vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:32.322893 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc9afae97ec4d6fabefad1a319e913c" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a"
I20240501 20:08:32.323019 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc9afae97ec4d6fabefad1a319e913c" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:32.323088 14947 raft_consensus.cc:3037] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.323194 15078 raft_consensus.cc:3037] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.324959 14947 raft_consensus.cc:2454] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 1.
I20240501 20:08:32.325241 15078 raft_consensus.cc:2454] T dcc9afae97ec4d6fabefad1a319e913c P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 1.
I20240501 20:08:32.325304 15141 leader_election.cc:302] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:32.325598 15942 raft_consensus.cc:2781] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:32.326035 15942 raft_consensus.cc:686] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Becoming Leader. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Running, Role: LEADER
I20240501 20:08:32.326699 15942 consensus_queue.cc:237] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:32.331162 14805 catalog_manager.cc:5484] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2177fb70b1324acb81b2bea7be19955d (127.14.106.195). New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } }
W20240501 20:08:32.343818 15124 tablet.cc:2369] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:08:32.404767 14993 tablet.cc:2369] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:32.475597 15941 raft_consensus.cc:482] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:32.475788 15941 raft_consensus.cc:504] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.477442 15941 leader_election.cc:288] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:32.482481 15942 raft_consensus.cc:482] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:32.482718 15942 raft_consensus.cc:504] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.483568 15942 leader_election.cc:288] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:32.484066 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" is_pre_election: true
I20240501 20:08:32.484372 15078 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 0.
I20240501 20:08:32.484397 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" is_pre_election: true
I20240501 20:08:32.484795 15077 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 0.
I20240501 20:08:32.485227 15274 leader_election.cc:302] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:32.485430 15144 leader_election.cc:302] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 4ed2ac58f951430a8c7e084f97d7a2a4; no voters:
I20240501 20:08:32.485649 15941 raft_consensus.cc:2781] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:32.485733 15942 raft_consensus.cc:2781] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:32.485762 15941 raft_consensus.cc:482] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:32.485842 15942 raft_consensus.cc:482] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:32.485980 15942 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.485985 15941 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.485893 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:32.488718 15942 raft_consensus.cc:504] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.488724 15941 raft_consensus.cc:504] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.489135 15208 raft_consensus.cc:2379] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e494ca2918594ba792c2d37d98b8c607 in current term 1: Already voted for candidate 2177fb70b1324acb81b2bea7be19955d in this term.
I20240501 20:08:32.489457 15941 leader_election.cc:288] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:32.490093 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:32.490272 15077 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:32.490429 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:32.490650 15208 raft_consensus.cc:2379] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e494ca2918594ba792c2d37d98b8c607 in current term 1: Already voted for candidate 2177fb70b1324acb81b2bea7be19955d in this term.
I20240501 20:08:32.491673 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:32.491797 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:32.492084 15338 raft_consensus.cc:2379] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2177fb70b1324acb81b2bea7be19955d in current term 1: Already voted for candidate e494ca2918594ba792c2d37d98b8c607 in this term.
I20240501 20:08:32.492372 15942 leader_election.cc:288] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 election: Requested vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:32.492678 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:32.492928 15337 raft_consensus.cc:2379] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2177fb70b1324acb81b2bea7be19955d in current term 1: Already voted for candidate e494ca2918594ba792c2d37d98b8c607 in this term.
I20240501 20:08:32.493130 15077 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 1.
I20240501 20:08:32.493309 15142 leader_election.cc:302] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d; no voters: 4ed2ac58f951430a8c7e084f97d7a2a4, e494ca2918594ba792c2d37d98b8c607
I20240501 20:08:32.493471 15274 leader_election.cc:302] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, e494ca2918594ba792c2d37d98b8c607; no voters: 2177fb70b1324acb81b2bea7be19955d
I20240501 20:08:32.493595 15942 raft_consensus.cc:2726] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:08:32.493837 15941 raft_consensus.cc:2781] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:32.494341 15941 raft_consensus.cc:686] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Becoming Leader. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Running, Role: LEADER
I20240501 20:08:32.494837 15941 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:32.499157 14805 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 reported cstate change: term changed from 0 to 1, leader changed from <none> to e494ca2918594ba792c2d37d98b8c607 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:32.695968 15943 consensus_queue.cc:1035] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:32.707600 15951 consensus_queue.cc:1035] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:32.731720 15942 consensus_queue.cc:1035] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:32.738214 15955 consensus_queue.cc:1035] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:32.874128 15941 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:32.879734 15941 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:32.908591 15961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:32.908915 15961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:32.916173 15961 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 268.854 ms
time per row: 0.134427 ms
I20240501 20:08:33.336943 14805 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60302:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:33.345351 14926 tablet_service.cc:1467] Processing CreateTablet for tablet 95e65bcdb5174beb91a9edb54d20a102 (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.345810 14926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e65bcdb5174beb91a9edb54d20a102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.345750 15187 tablet_service.cc:1467] Processing CreateTablet for tablet 95e65bcdb5174beb91a9edb54d20a102 (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.346213 15187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e65bcdb5174beb91a9edb54d20a102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.346689 15188 tablet_service.cc:1467] Processing CreateTablet for tablet 051f09e892c043269ed3e749c8f393fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.347088 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051f09e892c043269ed3e749c8f393fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.347076 14927 tablet_service.cc:1467] Processing CreateTablet for tablet 051f09e892c043269ed3e749c8f393fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.347446 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051f09e892c043269ed3e749c8f393fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.350025 15057 tablet_service.cc:1467] Processing CreateTablet for tablet 3931f8694a4a45ee81497f1c0b263d4d (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.350427 15057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3931f8694a4a45ee81497f1c0b263d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.352067 15318 tablet_service.cc:1467] Processing CreateTablet for tablet 95e65bcdb5174beb91a9edb54d20a102 (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.352478 15318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e65bcdb5174beb91a9edb54d20a102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.353191 14925 tablet_service.cc:1467] Processing CreateTablet for tablet 3931f8694a4a45ee81497f1c0b263d4d (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.353585 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3931f8694a4a45ee81497f1c0b263d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.358723 15186 tablet_service.cc:1467] Processing CreateTablet for tablet 3931f8694a4a45ee81497f1c0b263d4d (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.358992 15317 tablet_service.cc:1467] Processing CreateTablet for tablet 051f09e892c043269ed3e749c8f393fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=13166e6d76ec474fb27d6d635ad5704f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:33.359113 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3931f8694a4a45ee81497f1c0b263d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.359426 15317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051f09e892c043269ed3e749c8f393fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.364214 16025 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:33.366564 16025 tablet_bootstrap.cc:654] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.370316 16026 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:33.371248 16027 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:33.373330 16026 tablet_bootstrap.cc:654] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.373778 16027 tablet_bootstrap.cc:654] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.374980 16025 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:33.375027 16028 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:33.375201 16025 ts_tablet_manager.cc:1367] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 20:08:33.376305 16025 raft_consensus.cc:348] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.376695 16025 raft_consensus.cc:374] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.376875 16025 raft_consensus.cc:729] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.377266 16025 consensus_queue.cc:260] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.378062 16028 tablet_bootstrap.cc:654] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.378890 16025 ts_tablet_manager.cc:1398] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:08:33.379274 16025 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:33.382217 16025 tablet_bootstrap.cc:654] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.386201 16028 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:33.386405 16028 ts_tablet_manager.cc:1367] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20240501 20:08:33.386474 16026 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:33.386687 16026 ts_tablet_manager.cc:1367] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20240501 20:08:33.387267 16027 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:33.387383 16025 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:33.387280 16028 raft_consensus.cc:348] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.387478 16028 raft_consensus.cc:374] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.387503 16027 ts_tablet_manager.cc:1367] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20240501 20:08:33.387564 16025 ts_tablet_manager.cc:1367] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 20:08:33.387501 16026 raft_consensus.cc:348] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.387672 16028 raft_consensus.cc:729] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.387765 16026 raft_consensus.cc:374] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.387904 16026 raft_consensus.cc:729] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.387986 16028 consensus_queue.cc:260] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.388228 16026 consensus_queue.cc:260] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.388276 16025 raft_consensus.cc:348] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.388314 16027 raft_consensus.cc:348] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.388482 16025 raft_consensus.cc:374] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.388572 16025 raft_consensus.cc:729] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.388592 16027 raft_consensus.cc:374] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.388713 16027 raft_consensus.cc:729] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.388845 16025 consensus_queue.cc:260] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.389411 16025 ts_tablet_manager.cc:1398] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:08:33.389703 16025 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:33.388995 16027 consensus_queue.cc:260] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.390519 16027 ts_tablet_manager.cc:1398] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:08:33.391252 16025 tablet_bootstrap.cc:654] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.392153 16028 ts_tablet_manager.cc:1398] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:08:33.393155 16028 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:33.395153 16028 tablet_bootstrap.cc:654] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.397714 16026 ts_tablet_manager.cc:1398] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240501 20:08:33.398990 16026 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:33.402132 16026 tablet_bootstrap.cc:654] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.403154 16028 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:33.403379 16028 ts_tablet_manager.cc:1367] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20240501 20:08:33.403589 16025 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:33.403745 16025 ts_tablet_manager.cc:1367] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240501 20:08:33.404146 16028 raft_consensus.cc:348] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.404395 16026 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:33.404331 16025 raft_consensus.cc:348] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.404449 16028 raft_consensus.cc:374] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.404480 16025 raft_consensus.cc:374] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.404523 16028 raft_consensus.cc:729] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.404565 16026 ts_tablet_manager.cc:1367] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 20:08:33.404590 16025 raft_consensus.cc:729] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.404843 16025 consensus_queue.cc:260] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.404824 16028 consensus_queue.cc:260] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.405319 16025 ts_tablet_manager.cc:1398] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 20:08:33.405294 16026 raft_consensus.cc:348] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.405553 16026 raft_consensus.cc:374] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.405647 16026 raft_consensus.cc:729] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.405987 16028 ts_tablet_manager.cc:1398] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:08:33.405997 16026 consensus_queue.cc:260] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.406462 16026 ts_tablet_manager.cc:1398] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:33.406806 16026 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:33.408739 16026 tablet_bootstrap.cc:654] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.410943 16026 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:33.411113 16026 ts_tablet_manager.cc:1367] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:08:33.411832 16026 raft_consensus.cc:348] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.411978 16026 raft_consensus.cc:374] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.412041 16026 raft_consensus.cc:729] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.412216 16026 consensus_queue.cc:260] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.412663 16026 ts_tablet_manager.cc:1398] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:08:33.463577 16034 raft_consensus.cc:482] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:33.463747 16034 raft_consensus.cc:504] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.465338 16034 leader_election.cc:288] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:33.471828 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e65bcdb5174beb91a9edb54d20a102" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:33.471868 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e65bcdb5174beb91a9edb54d20a102" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:33.472082 15337 raft_consensus.cc:2454] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 0.
I20240501 20:08:33.472093 15208 raft_consensus.cc:2454] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 0.
I20240501 20:08:33.472546 15959 raft_consensus.cc:482] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:33.472573 14882 leader_election.cc:302] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:33.472684 15959 raft_consensus.cc:504] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.472977 16034 raft_consensus.cc:2781] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:33.473098 16034 raft_consensus.cc:482] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:33.473201 16034 raft_consensus.cc:3037] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.473632 15959 leader_election.cc:288] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:33.474469 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051f09e892c043269ed3e749c8f393fc" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:33.474726 15208 raft_consensus.cc:2454] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 0.
I20240501 20:08:33.475165 15273 leader_election.cc:302] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:33.475418 15959 raft_consensus.cc:2781] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:33.475524 15959 raft_consensus.cc:482] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:33.475613 15959 raft_consensus.cc:3037] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.476614 16034 raft_consensus.cc:504] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.477456 15959 raft_consensus.cc:504] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.477561 16034 leader_election.cc:288] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:33.478008 15959 leader_election.cc:288] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:33.478462 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e65bcdb5174beb91a9edb54d20a102" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:33.478490 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051f09e892c043269ed3e749c8f393fc" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:33.478662 15337 raft_consensus.cc:3037] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.478682 15208 raft_consensus.cc:3037] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.481405 15337 raft_consensus.cc:2454] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 1.
I20240501 20:08:33.481493 15208 raft_consensus.cc:2454] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 1.
I20240501 20:08:33.481808 15965 raft_consensus.cc:482] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:33.481925 15273 leader_election.cc:302] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:33.481963 15965 raft_consensus.cc:504] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.482544 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e65bcdb5174beb91a9edb54d20a102" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:33.482757 15208 raft_consensus.cc:3037] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.482893 15959 raft_consensus.cc:2781] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:33.483110 15959 raft_consensus.cc:686] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Becoming Leader. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Running, Role: LEADER
I20240501 20:08:33.483338 15965 leader_election.cc:288] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095)
I20240501 20:08:33.483989 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3931f8694a4a45ee81497f1c0b263d4d" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a" is_pre_election: true
I20240501 20:08:33.484100 14881 leader_election.cc:302] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:33.484239 14946 raft_consensus.cc:2454] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:33.484403 16034 raft_consensus.cc:2781] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:33.483422 15959 consensus_queue.cc:237] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:33.485405 15011 leader_election.cc:302] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:33.485901 15965 raft_consensus.cc:2781] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:33.486009 15965 raft_consensus.cc:482] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:33.486090 15965 raft_consensus.cc:3037] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.486335 15208 raft_consensus.cc:2454] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 1.
I20240501 20:08:33.488406 15965 raft_consensus.cc:504] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.489338 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051f09e892c043269ed3e749c8f393fc" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a" is_pre_election: true
I20240501 20:08:33.489526 14947 raft_consensus.cc:2454] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 0.
I20240501 20:08:33.489553 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051f09e892c043269ed3e749c8f393fc" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a"
I20240501 20:08:33.489826 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3931f8694a4a45ee81497f1c0b263d4d" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a"
I20240501 20:08:33.489881 14945 raft_consensus.cc:3037] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.489955 14946 raft_consensus.cc:3037] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.490201 14806 catalog_manager.cc:5484] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 reported cstate change: term changed from 0 to 1, leader changed from <none> to e494ca2918594ba792c2d37d98b8c607 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:33.492904 14945 raft_consensus.cc:2454] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 1.
I20240501 20:08:33.492978 14946 raft_consensus.cc:2454] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:33.493322 15011 leader_election.cc:302] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:33.493585 15951 raft_consensus.cc:2781] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:33.493729 15951 raft_consensus.cc:686] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Running, Role: LEADER
I20240501 20:08:33.494112 15951 consensus_queue.cc:237] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:33.494895 15965 leader_election.cc:288] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095)
I20240501 20:08:33.500373 16034 raft_consensus.cc:686] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 1 LEADER]: Becoming Leader. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Running, Role: LEADER
I20240501 20:08:33.500895 16034 consensus_queue.cc:237] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:33.502053 14806 catalog_manager.cc:5484] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:33.504794 14805 catalog_manager.cc:5484] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7decc3355b114404abddb7a10cc4358a (127.14.106.193). New cstate: current_term: 1 leader_uuid: "7decc3355b114404abddb7a10cc4358a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:33.507318 15207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3931f8694a4a45ee81497f1c0b263d4d" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:33.507558 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3931f8694a4a45ee81497f1c0b263d4d" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:33.507616 15207 raft_consensus.cc:3037] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.509526 15207 raft_consensus.cc:2454] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
W20240501 20:08:33.853391 16048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:33.853705 16048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:33.861109 16048 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:08:33.932644 16034 consensus_queue.cc:1035] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:33.942843 16043 consensus_queue.cc:1035] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:34.017984 14945 raft_consensus.cc:1261] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:34.018554 15959 consensus_queue.cc:1035] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:34.028234 15951 consensus_queue.cc:1035] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:34.030493 15207 raft_consensus.cc:1261] T 051f09e892c043269ed3e749c8f393fc P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:34.032011 15959 consensus_queue.cc:1035] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:34.060504 16075 mvcc.cc:205] Tried to move back new op lower bound from 7022977491008958464 to 7022977488843804672. Current Snapshot: MvccSnapshot[applied={T|T < 7022977491008958464}]
I20240501 20:08:34.080335 16073 mvcc.cc:205] Tried to move back new op lower bound from 7022977491008958464 to 7022977488843804672. Current Snapshot: MvccSnapshot[applied={T|T < 7022977491008958464}]
I20240501 20:08:34.091161 15205 raft_consensus.cc:1261] T 3931f8694a4a45ee81497f1c0b263d4d P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:34.093580 15951 consensus_queue.cc:1035] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:34.232666 16087 mvcc.cc:205] Tried to move back new op lower bound from 7022977491195764736 to 7022977488875327488. Current Snapshot: MvccSnapshot[applied={T|T < 7022977491195764736}]
INSERT report
rows total: 2000
time total: 345.997 ms
time per row: 0.172999 ms
I20240501 20:08:34.350302 14806 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60302:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:34.358932 15317 tablet_service.cc:1467] Processing CreateTablet for tablet c28b52cf6d9d4fa289de88ef97e08bee (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.359436 15317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b52cf6d9d4fa289de88ef97e08bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.359519 14926 tablet_service.cc:1467] Processing CreateTablet for tablet c28b52cf6d9d4fa289de88ef97e08bee (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.359926 14926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b52cf6d9d4fa289de88ef97e08bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.360904 15058 tablet_service.cc:1467] Processing CreateTablet for tablet 07797a04ba4b4ee3801855bfdbc1720d (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.360882 15057 tablet_service.cc:1467] Processing CreateTablet for tablet c28b52cf6d9d4fa289de88ef97e08bee (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.361405 15057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b52cf6d9d4fa289de88ef97e08bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.361604 15058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07797a04ba4b4ee3801855bfdbc1720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.363046 15056 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6758c002cc46f4bf960bff1196c457 (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.363471 15056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6758c002cc46f4bf960bff1196c457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.367408 15188 tablet_service.cc:1467] Processing CreateTablet for tablet 07797a04ba4b4ee3801855bfdbc1720d (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.367834 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07797a04ba4b4ee3801855bfdbc1720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.368041 14925 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6758c002cc46f4bf960bff1196c457 (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.368422 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6758c002cc46f4bf960bff1196c457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.371007 15318 tablet_service.cc:1467] Processing CreateTablet for tablet 07797a04ba4b4ee3801855bfdbc1720d (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.371364 15318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07797a04ba4b4ee3801855bfdbc1720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.375062 15316 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6758c002cc46f4bf960bff1196c457 (DEFAULT_TABLE table=rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:34.375438 15316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6758c002cc46f4bf960bff1196c457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.376029 16115 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:34.378401 16115 tablet_bootstrap.cc:654] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.380196 16116 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:34.381371 16115 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:34.381556 16115 ts_tablet_manager.cc:1367] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:08:34.382120 16116 tablet_bootstrap.cc:654] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.382787 16115 raft_consensus.cc:348] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.382999 16115 raft_consensus.cc:374] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.383157 16115 raft_consensus.cc:729] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.383499 16114 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:34.383438 16115 consensus_queue.cc:260] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.384104 16115 ts_tablet_manager.cc:1398] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:34.384438 16115 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:34.385138 16114 tablet_bootstrap.cc:654] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.386073 16115 tablet_bootstrap.cc:654] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.390663 16119 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:34.391183 16114 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:34.391884 16114 ts_tablet_manager.cc:1367] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240501 20:08:34.392488 16115 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:34.392688 16115 ts_tablet_manager.cc:1367] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:08:34.392791 16114 raft_consensus.cc:348] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.392978 16119 tablet_bootstrap.cc:654] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.393074 16114 raft_consensus.cc:374] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.393198 16114 raft_consensus.cc:729] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.393479 16114 consensus_queue.cc:260] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.394140 16114 ts_tablet_manager.cc:1398] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:34.394455 16114 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:34.394878 16116 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:34.395025 16116 ts_tablet_manager.cc:1367] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240501 20:08:34.395846 16116 raft_consensus.cc:348] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.396057 16116 raft_consensus.cc:374] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.396147 16116 raft_consensus.cc:729] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.396401 16116 consensus_queue.cc:260] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.393676 16115 raft_consensus.cc:348] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.396888 16115 raft_consensus.cc:374] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.396971 16115 raft_consensus.cc:729] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.397133 16116 ts_tablet_manager.cc:1398] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:34.397224 16115 consensus_queue.cc:260] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.397387 16116 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:34.397823 16115 ts_tablet_manager.cc:1398] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:08:34.398478 16115 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:34.398977 16116 tablet_bootstrap.cc:654] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.400115 16115 tablet_bootstrap.cc:654] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.400982 16114 tablet_bootstrap.cc:654] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.403492 16119 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:34.403744 16119 ts_tablet_manager.cc:1367] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240501 20:08:34.404829 16119 raft_consensus.cc:348] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.405172 16119 raft_consensus.cc:374] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.405328 16119 raft_consensus.cc:729] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.405722 16116 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:34.405930 16116 ts_tablet_manager.cc:1367] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20240501 20:08:34.405699 16119 consensus_queue.cc:260] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.406895 16116 raft_consensus.cc:348] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.407404 16119 ts_tablet_manager.cc:1398] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:08:34.408375 16115 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:34.408599 16116 raft_consensus.cc:374] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.408676 16115 ts_tablet_manager.cc:1367] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240501 20:08:34.408689 16116 raft_consensus.cc:729] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.408999 16114 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:34.408933 16116 consensus_queue.cc:260] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.409140 16114 ts_tablet_manager.cc:1367] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.005s
I20240501 20:08:34.409435 16116 ts_tablet_manager.cc:1398] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:34.409458 16115 raft_consensus.cc:348] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.409642 16115 raft_consensus.cc:374] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.409726 16115 raft_consensus.cc:729] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.409881 16114 raft_consensus.cc:348] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.409960 16115 consensus_queue.cc:260] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.410077 16114 raft_consensus.cc:374] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.410156 16114 raft_consensus.cc:729] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.410880 16114 consensus_queue.cc:260] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.411402 16114 ts_tablet_manager.cc:1398] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:34.411701 16114 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:34.413098 16115 ts_tablet_manager.cc:1398] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:08:34.413911 16114 tablet_bootstrap.cc:654] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:34.415704 16114 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:34.415828 16114 ts_tablet_manager.cc:1367] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:08:34.416368 16114 raft_consensus.cc:348] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.416505 16114 raft_consensus.cc:374] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:34.416599 16114 raft_consensus.cc:729] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.416785 16114 consensus_queue.cc:260] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.417155 16114 ts_tablet_manager.cc:1398] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:08:34.529507 16043 raft_consensus.cc:482] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:34.529740 16043 raft_consensus.cc:504] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.530544 16043 leader_election.cc:288] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:34.531188 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:34.531504 15337 raft_consensus.cc:2454] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 0.
I20240501 20:08:34.531934 14881 leader_election.cc:302] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:34.532243 16043 raft_consensus.cc:2781] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:34.532346 16043 raft_consensus.cc:482] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:34.532451 16043 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.535351 16043 raft_consensus.cc:504] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.536207 16043 leader_election.cc:288] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 election: Requested vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:34.536289 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:34.536537 15337 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.537168 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" is_pre_election: true
I20240501 20:08:34.537168 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:34.537446 15077 raft_consensus.cc:2454] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 0.
I20240501 20:08:34.538908 15337 raft_consensus.cc:2454] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 1.
I20240501 20:08:34.539300 14881 leader_election.cc:302] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607; no voters: 4ed2ac58f951430a8c7e084f97d7a2a4
I20240501 20:08:34.539599 16043 raft_consensus.cc:2781] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:34.539779 16043 raft_consensus.cc:686] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 1 LEADER]: Becoming Leader. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Running, Role: LEADER
I20240501 20:08:34.540063 16043 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.542798 14802 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7decc3355b114404abddb7a10cc4358a (127.14.106.193). New cstate: current_term: 1 leader_uuid: "7decc3355b114404abddb7a10cc4358a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:34.572909 16086 raft_consensus.cc:482] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:34.573086 16086 raft_consensus.cc:504] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.573792 16086 leader_election.cc:288] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:34.574292 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6758c002cc46f4bf960bff1196c457" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a" is_pre_election: true
I20240501 20:08:34.574362 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6758c002cc46f4bf960bff1196c457" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" is_pre_election: true
I20240501 20:08:34.574550 14947 raft_consensus.cc:2454] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 0.
I20240501 20:08:34.574589 15078 raft_consensus.cc:2454] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 0.
I20240501 20:08:34.575003 15271 leader_election.cc:302] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:34.575280 16086 raft_consensus.cc:2781] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:34.575397 16086 raft_consensus.cc:482] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:34.575496 16086 raft_consensus.cc:3037] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.577370 16086 raft_consensus.cc:504] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.577970 16086 leader_election.cc:288] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 election: Requested vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:34.578425 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6758c002cc46f4bf960bff1196c457" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:34.578492 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6758c002cc46f4bf960bff1196c457" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a"
I20240501 20:08:34.578610 15078 raft_consensus.cc:3037] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.578742 14947 raft_consensus.cc:3037] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.580454 15078 raft_consensus.cc:2454] T 9c6758c002cc46f4bf960bff1196c457 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 1.
I20240501 20:08:34.580509 14947 raft_consensus.cc:2454] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 1.
I20240501 20:08:34.580834 15271 leader_election.cc:302] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:34.581247 16086 raft_consensus.cc:2781] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:34.581398 16086 raft_consensus.cc:686] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Becoming Leader. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Running, Role: LEADER
I20240501 20:08:34.581651 16086 consensus_queue.cc:237] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:34.584318 14802 catalog_manager.cc:5484] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 reported cstate change: term changed from 0 to 1, leader changed from <none> to e494ca2918594ba792c2d37d98b8c607 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:34.616717 15955 raft_consensus.cc:482] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:34.616967 15955 raft_consensus.cc:504] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.617599 15955 leader_election.cc:288] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:34.618191 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07797a04ba4b4ee3801855bfdbc1720d" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" is_pre_election: true
I20240501 20:08:34.618185 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07797a04ba4b4ee3801855bfdbc1720d" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:34.618407 15337 raft_consensus.cc:2454] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 0.
I20240501 20:08:34.618392 15078 raft_consensus.cc:2454] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 0.
I20240501 20:08:34.618854 15144 leader_election.cc:302] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 4ed2ac58f951430a8c7e084f97d7a2a4; no voters:
I20240501 20:08:34.619158 15955 raft_consensus.cc:2781] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:34.619240 15955 raft_consensus.cc:482] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:34.619364 15955 raft_consensus.cc:3037] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.621255 15955 raft_consensus.cc:504] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.621825 15955 leader_election.cc:288] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 election: Requested vote from peers 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:34.622315 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07797a04ba4b4ee3801855bfdbc1720d" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:34.622354 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07797a04ba4b4ee3801855bfdbc1720d" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:34.622489 15337 raft_consensus.cc:3037] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.622542 15078 raft_consensus.cc:3037] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.624456 15078 raft_consensus.cc:2454] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 1.
I20240501 20:08:34.624495 15337 raft_consensus.cc:2454] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 1.
I20240501 20:08:34.624873 15144 leader_election.cc:302] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 4ed2ac58f951430a8c7e084f97d7a2a4; no voters:
I20240501 20:08:34.625180 15955 raft_consensus.cc:2781] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:34.625567 15955 raft_consensus.cc:686] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Becoming Leader. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Running, Role: LEADER
I20240501 20:08:34.625862 15955 consensus_queue.cc:237] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:34.628541 14802 catalog_manager.cc:5484] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2177fb70b1324acb81b2bea7be19955d (127.14.106.195). New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:34.953859 16043 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:34.959695 15078 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:34.963321 16043 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:35.011518 16132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:35.011927 16132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:08:35.018345 15955 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:35.023304 16132 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:08:35.026576 15955 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 20:08:35.065716 16086 consensus_queue.cc:1035] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:35.071043 15959 consensus_queue.cc:1035] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 256.016 ms
time per row: 0.128008 ms
I20240501 20:08:35.425895 14802 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60302:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:35.434535 15188 tablet_service.cc:1467] Processing CreateTablet for tablet 2a672748de3e4e60bd950d6cffbcfd08 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.435022 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a672748de3e4e60bd950d6cffbcfd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.435088 14925 tablet_service.cc:1467] Processing CreateTablet for tablet 2a672748de3e4e60bd950d6cffbcfd08 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.435322 15317 tablet_service.cc:1467] Processing CreateTablet for tablet 3e873682369f4d4f93dbe5fb3b369d1f (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.435529 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a672748de3e4e60bd950d6cffbcfd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.435691 15317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e873682369f4d4f93dbe5fb3b369d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.439388 15316 tablet_service.cc:1467] Processing CreateTablet for tablet efbf0e49f78a4ff099786f47eedc3c8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.439633 15056 tablet_service.cc:1467] Processing CreateTablet for tablet 2a672748de3e4e60bd950d6cffbcfd08 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.439785 15316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbf0e49f78a4ff099786f47eedc3c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.440042 15056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a672748de3e4e60bd950d6cffbcfd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.440701 15057 tablet_service.cc:1467] Processing CreateTablet for tablet efbf0e49f78a4ff099786f47eedc3c8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.441062 15057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbf0e49f78a4ff099786f47eedc3c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.441891 15187 tablet_service.cc:1467] Processing CreateTablet for tablet 3e873682369f4d4f93dbe5fb3b369d1f (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.442286 15187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e873682369f4d4f93dbe5fb3b369d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.447082 15186 tablet_service.cc:1467] Processing CreateTablet for tablet efbf0e49f78a4ff099786f47eedc3c8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.447579 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbf0e49f78a4ff099786f47eedc3c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.450698 16197 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:35.452838 16197 tablet_bootstrap.cc:654] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.455936 14926 tablet_service.cc:1467] Processing CreateTablet for tablet 3e873682369f4d4f93dbe5fb3b369d1f (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.456161 16197 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:35.456303 16197 ts_tablet_manager.cc:1367] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 20:08:35.456321 14926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e873682369f4d4f93dbe5fb3b369d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.457108 16197 raft_consensus.cc:348] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.457322 16197 raft_consensus.cc:374] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.457407 16197 raft_consensus.cc:729] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.457617 16197 consensus_queue.cc:260] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.458873 16197 ts_tablet_manager.cc:1398] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:08:35.459157 16197 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:35.461345 16197 tablet_bootstrap.cc:654] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.465605 16199 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:35.466473 16196 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:35.468997 16196 tablet_bootstrap.cc:654] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.470028 16197 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:35.470314 16197 ts_tablet_manager.cc:1367] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 20:08:35.470746 16199 tablet_bootstrap.cc:654] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.471453 16197 raft_consensus.cc:348] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.471822 16197 raft_consensus.cc:374] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.471963 16197 raft_consensus.cc:729] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.472359 16197 consensus_queue.cc:260] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.473994 16197 ts_tablet_manager.cc:1398] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:08:35.476622 16199 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:35.476871 16199 ts_tablet_manager.cc:1367] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20240501 20:08:35.477950 16199 raft_consensus.cc:348] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.478289 16199 raft_consensus.cc:374] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.478440 16199 raft_consensus.cc:729] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.478778 16199 consensus_queue.cc:260] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.479485 16199 ts_tablet_manager.cc:1398] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:35.479799 16199 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:35.481609 16199 tablet_bootstrap.cc:654] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.483840 16196 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:35.484004 16196 ts_tablet_manager.cc:1367] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20240501 20:08:35.484802 16196 raft_consensus.cc:348] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.485035 16196 raft_consensus.cc:374] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.485152 16196 raft_consensus.cc:729] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.485388 16196 consensus_queue.cc:260] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.485978 16196 ts_tablet_manager.cc:1398] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:08:35.486243 16196 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:35.487530 16196 tablet_bootstrap.cc:654] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.488363 16203 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:35.488677 16199 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:35.488840 16199 ts_tablet_manager.cc:1367] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240501 20:08:35.489544 16199 raft_consensus.cc:348] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.489769 16199 raft_consensus.cc:374] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.489894 16199 raft_consensus.cc:729] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.490170 16199 consensus_queue.cc:260] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.490525 16203 tablet_bootstrap.cc:654] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.491824 16199 ts_tablet_manager.cc:1398] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:08:35.495637 16203 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:35.495666 16196 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:35.495762 16203 ts_tablet_manager.cc:1367] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:08:35.495816 16196 ts_tablet_manager.cc:1367] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240501 20:08:35.496414 16196 raft_consensus.cc:348] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.496471 16203 raft_consensus.cc:348] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.496582 16196 raft_consensus.cc:374] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.496635 16203 raft_consensus.cc:374] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.496665 16196 raft_consensus.cc:729] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.496730 16203 raft_consensus.cc:729] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.496908 16196 consensus_queue.cc:260] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.496973 16203 consensus_queue.cc:260] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.497411 16196 ts_tablet_manager.cc:1398] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 20:08:35.497524 16203 ts_tablet_manager.cc:1398] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:35.497684 16196 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:35.497817 16203 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:35.499258 16196 tablet_bootstrap.cc:654] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.499326 16203 tablet_bootstrap.cc:654] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.500936 16196 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:35.501078 16196 ts_tablet_manager.cc:1367] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:35.501106 16203 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:35.501248 16203 ts_tablet_manager.cc:1367] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:35.501865 16203 raft_consensus.cc:348] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.501856 16196 raft_consensus.cc:348] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.502032 16196 raft_consensus.cc:374] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.502068 16203 raft_consensus.cc:374] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.502146 16196 raft_consensus.cc:729] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.502190 16203 raft_consensus.cc:729] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.502403 16196 consensus_queue.cc:260] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.502468 16203 consensus_queue.cc:260] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.503041 16196 ts_tablet_manager.cc:1398] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:08:35.503139 16203 ts_tablet_manager.cc:1398] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:08:35.618867 16143 raft_consensus.cc:482] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:35.619064 16143 raft_consensus.cc:504] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.619746 16143 leader_election.cc:288] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:35.620196 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" is_pre_election: true
I20240501 20:08:35.620326 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:35.620488 15077 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 0.
I20240501 20:08:35.620568 15338 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 0.
I20240501 20:08:35.620951 15144 leader_election.cc:302] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 4ed2ac58f951430a8c7e084f97d7a2a4; no voters:
I20240501 20:08:35.621243 16143 raft_consensus.cc:2781] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:35.621362 16143 raft_consensus.cc:482] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:35.621431 16143 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.623282 16143 raft_consensus.cc:504] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.623796 16143 leader_election.cc:288] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 election: Requested vote from peers 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:35.624190 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:35.624289 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:35.624420 15077 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.624507 15338 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.627233 15077 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 1.
I20240501 20:08:35.627233 15338 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 1.
I20240501 20:08:35.627612 15142 leader_election.cc:302] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:35.627889 16143 raft_consensus.cc:2781] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:35.628046 16143 raft_consensus.cc:686] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Becoming Leader. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Running, Role: LEADER
I20240501 20:08:35.628335 16143 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:08:35.631587 16043 raft_consensus.cc:482] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:35.631755 16043 raft_consensus.cc:504] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.631937 14803 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2177fb70b1324acb81b2bea7be19955d (127.14.106.195). New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:35.632519 16043 leader_election.cc:288] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:35.632872 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e873682369f4d4f93dbe5fb3b369d1f" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:35.632923 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e873682369f4d4f93dbe5fb3b369d1f" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:35.633060 15206 raft_consensus.cc:2454] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 0.
I20240501 20:08:35.633118 15338 raft_consensus.cc:2454] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 0.
I20240501 20:08:35.633456 14882 leader_election.cc:302] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:35.633733 16043 raft_consensus.cc:2781] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:35.633821 16043 raft_consensus.cc:482] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:35.633874 16043 raft_consensus.cc:3037] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.635944 16043 raft_consensus.cc:504] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.636581 16043 leader_election.cc:288] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:35.637022 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e873682369f4d4f93dbe5fb3b369d1f" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:35.637085 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e873682369f4d4f93dbe5fb3b369d1f" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:35.637219 15338 raft_consensus.cc:3037] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.637292 15206 raft_consensus.cc:3037] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.640075 15338 raft_consensus.cc:2454] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 1.
I20240501 20:08:35.640065 15206 raft_consensus.cc:2454] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 1.
I20240501 20:08:35.640455 14881 leader_election.cc:302] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:35.640777 16043 raft_consensus.cc:2781] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:35.640900 16043 raft_consensus.cc:686] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 1 LEADER]: Becoming Leader. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Running, Role: LEADER
I20240501 20:08:35.641152 16043 consensus_queue.cc:237] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:35.643679 14803 catalog_manager.cc:5484] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7decc3355b114404abddb7a10cc4358a (127.14.106.193). New cstate: current_term: 1 leader_uuid: "7decc3355b114404abddb7a10cc4358a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:35.693554 16107 raft_consensus.cc:482] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:35.693738 16107 raft_consensus.cc:504] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.694459 16107 leader_election.cc:288] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:35.694886 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a672748de3e4e60bd950d6cffbcfd08" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:35.694957 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a672748de3e4e60bd950d6cffbcfd08" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a" is_pre_election: true
I20240501 20:08:35.695108 15206 raft_consensus.cc:2454] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:35.695225 14947 raft_consensus.cc:2454] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:35.695521 15013 leader_election.cc:302] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 4ed2ac58f951430a8c7e084f97d7a2a4; no voters:
I20240501 20:08:35.695778 16107 raft_consensus.cc:2781] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:35.695888 16107 raft_consensus.cc:482] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:35.695978 16107 raft_consensus.cc:3037] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.698078 16107 raft_consensus.cc:504] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.698612 16107 leader_election.cc:288] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:35.698971 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a672748de3e4e60bd950d6cffbcfd08" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:35.699075 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a672748de3e4e60bd950d6cffbcfd08" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a"
I20240501 20:08:35.699141 15206 raft_consensus.cc:3037] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.699277 14947 raft_consensus.cc:3037] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.700933 15206 raft_consensus.cc:2454] T 2a672748de3e4e60bd950d6cffbcfd08 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:35.701038 14947 raft_consensus.cc:2454] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:35.701452 15011 leader_election.cc:302] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:35.701740 16107 raft_consensus.cc:2781] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:35.702154 16107 raft_consensus.cc:686] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Running, Role: LEADER
I20240501 20:08:35.702431 16107 consensus_queue.cc:237] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:35.705065 14803 catalog_manager.cc:5484] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } }
W20240501 20:08:36.077970 16213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:36.078310 16213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:36.085356 16213 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 20:08:36.159091 16143 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:36.179867 16143 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:36.208917 16043 consensus_queue.cc:1035] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:36.237511 16107 consensus_queue.cc:1035] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:36.277901 16260 consensus_queue.cc:1035] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:36.299454 16255 consensus_queue.cc:1035] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 270.762 ms
time per row: 0.135381 ms
I20240501 20:08:36.505349 14803 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60302:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:36.513797 14926 tablet_service.cc:1467] Processing CreateTablet for tablet 6513ee10256249b8976bfeb4cf0bdbb7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.513778 15188 tablet_service.cc:1467] Processing CreateTablet for tablet 6513ee10256249b8976bfeb4cf0bdbb7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.514254 14926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6513ee10256249b8976bfeb4cf0bdbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.514242 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6513ee10256249b8976bfeb4cf0bdbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.514580 15317 tablet_service.cc:1467] Processing CreateTablet for tablet bd92dbe7ec734430a5ab62db5b827849 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.515069 15317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd92dbe7ec734430a5ab62db5b827849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.515437 15187 tablet_service.cc:1467] Processing CreateTablet for tablet 7e54c2e99947456988c55a7e5c5c4f53 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.516139 15187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e54c2e99947456988c55a7e5c5c4f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.517206 15056 tablet_service.cc:1467] Processing CreateTablet for tablet 6513ee10256249b8976bfeb4cf0bdbb7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.517607 15056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6513ee10256249b8976bfeb4cf0bdbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.521497 15057 tablet_service.cc:1467] Processing CreateTablet for tablet bd92dbe7ec734430a5ab62db5b827849 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.521908 15057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd92dbe7ec734430a5ab62db5b827849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.522114 14925 tablet_service.cc:1467] Processing CreateTablet for tablet bd92dbe7ec734430a5ab62db5b827849 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.522604 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd92dbe7ec734430a5ab62db5b827849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.524385 16281 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:36.526556 16281 tablet_bootstrap.cc:654] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.527989 14927 tablet_service.cc:1467] Processing CreateTablet for tablet 7e54c2e99947456988c55a7e5c5c4f53 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.528338 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e54c2e99947456988c55a7e5c5c4f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.537948 15317 tablet_service.cc:1467] Processing CreateTablet for tablet 7e54c2e99947456988c55a7e5c5c4f53 (DEFAULT_TABLE table=rebalance_test_table_4 [id=149741be5950446bae57a1d52109ba6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:36.539494 15317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e54c2e99947456988c55a7e5c5c4f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:36.540753 16284 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:36.540884 16280 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:36.543737 16284 tablet_bootstrap.cc:654] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.545274 16281 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:36.545411 16281 ts_tablet_manager.cc:1367] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20240501 20:08:36.546209 16281 raft_consensus.cc:348] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.546424 16281 raft_consensus.cc:374] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.546550 16281 raft_consensus.cc:729] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.547263 16282 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:36.547482 16280 tablet_bootstrap.cc:654] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.549543 16282 tablet_bootstrap.cc:654] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.549813 16281 consensus_queue.cc:260] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.550365 16281 ts_tablet_manager.cc:1398] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:08:36.552362 16284 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:36.552549 16284 ts_tablet_manager.cc:1367] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240501 20:08:36.553450 16284 raft_consensus.cc:348] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.553782 16284 raft_consensus.cc:374] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.553928 16284 raft_consensus.cc:729] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.554216 16284 consensus_queue.cc:260] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.554448 16281 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607: Bootstrap starting.
I20240501 20:08:36.555483 16284 ts_tablet_manager.cc:1398] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:36.555804 16284 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:36.556059 16281 tablet_bootstrap.cc:654] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.558388 16284 tablet_bootstrap.cc:654] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.560838 16282 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:36.561067 16282 ts_tablet_manager.cc:1367] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240501 20:08:36.561353 16280 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:36.561493 16280 ts_tablet_manager.cc:1367] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240501 20:08:36.562129 16282 raft_consensus.cc:348] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.562402 16282 raft_consensus.cc:374] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.562311 16280 raft_consensus.cc:348] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.562497 16280 raft_consensus.cc:374] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.562506 16282 raft_consensus.cc:729] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.562575 16280 raft_consensus.cc:729] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.562799 16280 consensus_queue.cc:260] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.562795 16282 consensus_queue.cc:260] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.563357 16280 ts_tablet_manager.cc:1398] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:36.563524 16282 ts_tablet_manager.cc:1398] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:36.563627 16280 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d: Bootstrap starting.
I20240501 20:08:36.563848 16282 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4: Bootstrap starting.
I20240501 20:08:36.565893 16282 tablet_bootstrap.cc:654] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.570564 16280 tablet_bootstrap.cc:654] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.571317 16281 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607: No bootstrap required, opened a new log
I20240501 20:08:36.571465 16281 ts_tablet_manager.cc:1367] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20240501 20:08:36.571967 16284 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:36.572161 16284 ts_tablet_manager.cc:1367] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.011s
I20240501 20:08:36.572158 16281 raft_consensus.cc:348] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.572366 16281 raft_consensus.cc:374] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.572423 16281 raft_consensus.cc:729] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.572628 16281 consensus_queue.cc:260] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.573127 16281 ts_tablet_manager.cc:1398] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:08:36.573186 16284 raft_consensus.cc:348] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.573485 16284 raft_consensus.cc:374] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.573645 16284 raft_consensus.cc:729] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.574069 16284 consensus_queue.cc:260] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.575623 16284 ts_tablet_manager.cc:1398] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 20:08:36.575982 16284 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:08:36.578876 16284 tablet_bootstrap.cc:654] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:36.585477 16280 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d: No bootstrap required, opened a new log
I20240501 20:08:36.585470 16282 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4: No bootstrap required, opened a new log
I20240501 20:08:36.585637 16282 ts_tablet_manager.cc:1367] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.009s
I20240501 20:08:36.585649 16280 ts_tablet_manager.cc:1367] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20240501 20:08:36.586369 16284 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: No bootstrap required, opened a new log
I20240501 20:08:36.586498 16284 ts_tablet_manager.cc:1367] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20240501 20:08:36.586480 16280 raft_consensus.cc:348] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.586498 16282 raft_consensus.cc:348] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.586792 16280 raft_consensus.cc:374] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.586814 16282 raft_consensus.cc:374] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.586887 16282 raft_consensus.cc:729] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.586930 16280 raft_consensus.cc:729] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.587142 16282 consensus_queue.cc:260] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.587180 16280 consensus_queue.cc:260] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.587255 16284 raft_consensus.cc:348] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.587800 16280 ts_tablet_manager.cc:1398] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:36.587806 16282 ts_tablet_manager.cc:1398] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:36.587430 16284 raft_consensus.cc:374] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:36.588073 16284 raft_consensus.cc:729] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:08:36.588294 16284 consensus_queue.cc:260] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.588754 16284 ts_tablet_manager.cc:1398] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:36.610787 16043 raft_consensus.cc:482] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:36.610934 16043 raft_consensus.cc:504] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.611533 16043 leader_election.cc:288] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:36.611883 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e54c2e99947456988c55a7e5c5c4f53" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:36.612002 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e54c2e99947456988c55a7e5c5c4f53" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:36.612128 15208 raft_consensus.cc:2454] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 0.
I20240501 20:08:36.612251 15337 raft_consensus.cc:2454] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 0.
I20240501 20:08:36.612533 14882 leader_election.cc:302] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:36.612804 16043 raft_consensus.cc:2781] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:36.612915 16043 raft_consensus.cc:482] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:36.613003 16043 raft_consensus.cc:3037] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.614989 16043 raft_consensus.cc:504] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.615490 16043 leader_election.cc:288] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:36.615870 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e54c2e99947456988c55a7e5c5c4f53" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:36.615893 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e54c2e99947456988c55a7e5c5c4f53" candidate_uuid: "7decc3355b114404abddb7a10cc4358a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:36.616040 15208 raft_consensus.cc:3037] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.616062 15337 raft_consensus.cc:3037] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.617735 15208 raft_consensus.cc:2454] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 1.
I20240501 20:08:36.617794 15337 raft_consensus.cc:2454] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7decc3355b114404abddb7a10cc4358a in term 1.
I20240501 20:08:36.618077 14882 leader_election.cc:302] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:36.618448 16043 raft_consensus.cc:2781] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:36.618676 16043 raft_consensus.cc:686] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 1 LEADER]: Becoming Leader. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Running, Role: LEADER
I20240501 20:08:36.619032 16043 consensus_queue.cc:237] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.621706 14802 catalog_manager.cc:5484] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7decc3355b114404abddb7a10cc4358a (127.14.106.193). New cstate: current_term: 1 leader_uuid: "7decc3355b114404abddb7a10cc4358a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:36.640600 16255 raft_consensus.cc:482] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:36.640815 16255 raft_consensus.cc:504] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.641547 16255 leader_election.cc:288] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:36.641912 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6513ee10256249b8976bfeb4cf0bdbb7" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:36.642073 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6513ee10256249b8976bfeb4cf0bdbb7" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a" is_pre_election: true
I20240501 20:08:36.642155 15208 raft_consensus.cc:2454] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:36.642305 14946 raft_consensus.cc:2454] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:36.642647 15013 leader_election.cc:302] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 4ed2ac58f951430a8c7e084f97d7a2a4; no voters:
I20240501 20:08:36.643015 16255 raft_consensus.cc:2781] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:36.643131 16255 raft_consensus.cc:482] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:36.643270 16255 raft_consensus.cc:3037] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.645293 16255 raft_consensus.cc:504] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.645925 16255 leader_election.cc:288] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:36.646255 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6513ee10256249b8976bfeb4cf0bdbb7" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:36.646373 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6513ee10256249b8976bfeb4cf0bdbb7" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a"
I20240501 20:08:36.646442 15208 raft_consensus.cc:3037] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.646577 14946 raft_consensus.cc:3037] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.649344 15208 raft_consensus.cc:2454] T 6513ee10256249b8976bfeb4cf0bdbb7 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:36.649456 14946 raft_consensus.cc:2454] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:36.649909 15011 leader_election.cc:302] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:36.650161 16255 raft_consensus.cc:2781] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:36.650362 16255 raft_consensus.cc:686] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Running, Role: LEADER
I20240501 20:08:36.650688 16255 consensus_queue.cc:237] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:08:36.653468 14802 catalog_manager.cc:5484] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:36.675787 16255 raft_consensus.cc:482] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:36.675932 16255 raft_consensus.cc:504] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.676497 16255 leader_election.cc:288] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:36.676913 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd92dbe7ec734430a5ab62db5b827849" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:36.676962 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd92dbe7ec734430a5ab62db5b827849" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a" is_pre_election: true
I20240501 20:08:36.677158 14946 raft_consensus.cc:2454] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:36.677145 15337 raft_consensus.cc:2454] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 0.
I20240501 20:08:36.677565 15011 leader_election.cc:302] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, 7decc3355b114404abddb7a10cc4358a; no voters:
I20240501 20:08:36.677852 16255 raft_consensus.cc:2781] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:36.677963 16255 raft_consensus.cc:482] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:36.678030 16255 raft_consensus.cc:3037] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.679841 16255 raft_consensus.cc:504] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.680395 16255 leader_election.cc:288] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Requested vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:36.680830 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd92dbe7ec734430a5ab62db5b827849" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7decc3355b114404abddb7a10cc4358a"
I20240501 20:08:36.680912 15337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd92dbe7ec734430a5ab62db5b827849" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:36.681032 14946 raft_consensus.cc:3037] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.681099 15337 raft_consensus.cc:3037] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:36.683815 14946 raft_consensus.cc:2454] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:36.683825 15337 raft_consensus.cc:2454] T bd92dbe7ec734430a5ab62db5b827849 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
I20240501 20:08:36.684242 15012 leader_election.cc:302] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:36.684520 16255 raft_consensus.cc:2781] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:36.684635 16255 raft_consensus.cc:686] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Running, Role: LEADER
I20240501 20:08:36.684916 16255 consensus_queue.cc:237] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:36.687744 14802 catalog_manager.cc:5484] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:37.006856 16043 consensus_queue.cc:1035] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:37.015648 16043 consensus_queue.cc:1035] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:37.051338 16296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:37.051640 16296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:37.059427 16296 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 20:08:37.103655 16255 consensus_queue.cc:1035] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:37.132575 16273 consensus_queue.cc:1035] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:37.136698 16273 consensus_queue.cc:1035] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:37.159785 16255 consensus_queue.cc:1035] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 208.761 ms
time per row: 0.10438 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:08:38.370704 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:35831
--local_ip_for_outbound_sockets=127.14.106.197
--tserver_master_addrs=127.14.106.254:34075
--webserver_port=33771
--webserver_interface=127.14.106.197
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 20:08:38.705539 16357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:38.705920 16357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:38.706077 16357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:38.716148 16357 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:38.716308 16357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:38.716514 16357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:08:38.731700 16357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:35831
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=33771
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:38.733500 16357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:38.735873 16357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:38.750563 16365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:38.751276 16366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:38.752220 16357 server_base.cc:1034] running on GCE node
W20240501 20:08:38.752933 16368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:38.753566 16357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:38.754493 16357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:38.755688 16357 hybrid_clock.cc:648] HybridClock initialized: now 1714594118755630 us; error 54 us; skew 500 ppm
I20240501 20:08:38.756023 16357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:38.758339 16357 webserver.cc:460] Webserver started at http://127.14.106.197:33771/ using document root <none> and password file <none>
I20240501 20:08:38.759092 16357 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:38.759218 16357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:38.764144 16357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:08:38.766781 16374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:38.768033 16357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:08:38.768177 16357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "250badc514c94bcdbe2ace1b62a0874f"
format_stamp: "Formatted at 2024-05-01 20:08:26 on dist-test-slave-pg4x"
I20240501 20:08:38.768975 16357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:38.786007 16357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:38.787024 16357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:38.787309 16357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:38.788280 16357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:38.789683 16357 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:38.789754 16357 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:38.789878 16357 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:38.789927 16357 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:38.825093 16357 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:35831
I20240501 20:08:38.825134 16486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:35831 every 8 connection(s)
I20240501 20:08:38.826442 16357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:38.831072 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 16357
I20240501 20:08:38.831631 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.198:38031
--local_ip_for_outbound_sockets=127.14.106.198
--tserver_master_addrs=127.14.106.254:34075
--webserver_port=33963
--webserver_interface=127.14.106.198
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:38.839751 16487 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:38.840085 16487 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:38.841064 16487 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:38.842700 14803 ts_manager.cc:194] Re-registered known tserver with Master: 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197:35831)
I20240501 20:08:38.843490 14803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:44295
W20240501 20:08:39.165376 16491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:39.165645 16491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:39.165800 16491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:39.173734 16491 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:39.173880 16491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:39.174007 16491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.198
I20240501 20:08:39.182914 16491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.198:38031
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.106.198
--webserver_port=33963
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.198
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:39.184135 16491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:39.185739 16491 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:39.198341 16496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:39.199149 16497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:39.199839 16499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:39.200719 16491 server_base.cc:1034] running on GCE node
I20240501 20:08:39.201308 16491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:39.202369 16491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:39.203593 16491 hybrid_clock.cc:648] HybridClock initialized: now 1714594119203515 us; error 67 us; skew 500 ppm
I20240501 20:08:39.203898 16491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:39.206051 16491 webserver.cc:460] Webserver started at http://127.14.106.198:33963/ using document root <none> and password file <none>
I20240501 20:08:39.206838 16491 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:39.206969 16491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:39.212091 16491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:39.214991 16505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:39.216318 16491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:08:39.216476 16491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "98d6b3fd85674334a24cd273bf525324"
format_stamp: "Formatted at 2024-05-01 20:08:26 on dist-test-slave-pg4x"
I20240501 20:08:39.217281 16491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:39.229800 16491 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:39.230485 16491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:39.230803 16491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:39.231632 16491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:39.233003 16491 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:39.233080 16491 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:39.233206 16491 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:39.233265 16491 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:39.269387 16491 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.198:38031
I20240501 20:08:39.269446 16617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.198:38031 every 8 connection(s)
I20240501 20:08:39.270865 16491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:08:39.280164 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 16491
I20240501 20:08:39.280783 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.199:38431
--local_ip_for_outbound_sockets=127.14.106.199
--tserver_master_addrs=127.14.106.254:34075
--webserver_port=45985
--webserver_interface=127.14.106.199
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:39.284029 16618 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:39.284361 16618 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:39.285318 16618 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:39.286808 14803 ts_manager.cc:194] Re-registered known tserver with Master: 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031)
I20240501 20:08:39.287549 14803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.198:39665
W20240501 20:08:39.620946 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:39.621255 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:39.621423 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:39.628806 16621 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:39.628918 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:39.629074 16621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.199
I20240501 20:08:39.638024 16621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.199:38431
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.106.199
--webserver_port=45985
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.199
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:39.639313 16621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:39.640985 16621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:39.655236 16630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:39.656086 16628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:39.656747 16627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:39.657577 16621 server_base.cc:1034] running on GCE node
I20240501 20:08:39.658147 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:39.659137 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:39.660348 16621 hybrid_clock.cc:648] HybridClock initialized: now 1714594119660303 us; error 40 us; skew 500 ppm
I20240501 20:08:39.660645 16621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:39.663136 16621 webserver.cc:460] Webserver started at http://127.14.106.199:45985/ using document root <none> and password file <none>
I20240501 20:08:39.663834 16621 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:39.663957 16621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:39.668967 16621 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:08:39.671694 16636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:39.673094 16621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:39.673242 16621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "8dddd9d9b8fb47cb944ad4179152a915"
format_stamp: "Formatted at 2024-05-01 20:08:27 on dist-test-slave-pg4x"
I20240501 20:08:39.674064 16621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:39.686756 16621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:39.687505 16621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:39.687772 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:39.688674 16621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:39.690188 16621 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:39.690282 16621 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:39.690407 16621 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:39.690461 16621 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:39.723701 16621 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.199:38431
I20240501 20:08:39.723721 16748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.199:38431 every 8 connection(s)
I20240501 20:08:39.725410 16621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:08:39.729506 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 16621
I20240501 20:08:39.730175 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.200:40973
--local_ip_for_outbound_sockets=127.14.106.200
--tserver_master_addrs=127.14.106.254:34075
--webserver_port=41031
--webserver_interface=127.14.106.200
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:08:39.737797 16749 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:39.738126 16749 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:39.739153 16749 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:39.740702 14803 ts_manager.cc:194] Re-registered known tserver with Master: 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199:38431)
I20240501 20:08:39.741978 14803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.199:43353
I20240501 20:08:39.846506 16487 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
W20240501 20:08:40.068764 16752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:40.069054 16752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:40.069211 16752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:40.076740 16752 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:40.076877 16752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:40.077030 16752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.200
I20240501 20:08:40.085799 16752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.200:40973
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.106.200
--webserver_port=41031
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.200
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:40.087164 16752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:40.088876 16752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:40.101518 16757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:40.102389 16758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:40.103106 16760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:40.103936 16752 server_base.cc:1034] running on GCE node
I20240501 20:08:40.104461 16752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:40.105322 16752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:40.106519 16752 hybrid_clock.cc:648] HybridClock initialized: now 1714594120106465 us; error 47 us; skew 500 ppm
I20240501 20:08:40.106848 16752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:40.109097 16752 webserver.cc:460] Webserver started at http://127.14.106.200:41031/ using document root <none> and password file <none>
I20240501 20:08:40.109771 16752 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:40.109910 16752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:40.115180 16752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:08:40.117877 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.119200 16752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:40.119338 16752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0d9d84f0f3224d1a8fbaedd1021d730f"
format_stamp: "Formatted at 2024-05-01 20:08:27 on dist-test-slave-pg4x"
I20240501 20:08:40.120137 16752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:40.143625 16752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:40.144418 16752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:40.144703 16752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:40.145617 16752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:40.147118 16752 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:40.147198 16752 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.147327 16752 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:40.147385 16752 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.183115 16752 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.200:40973
I20240501 20:08:40.183167 16878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.200:40973 every 8 connection(s)
I20240501 20:08:40.184996 16752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:08:40.187817 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 16752
I20240501 20:08:40.200104 16879 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:08:40.200377 16879 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:40.201208 16879 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:08:40.202700 14803 ts_manager.cc:194] Re-registered known tserver with Master: 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973)
I20240501 20:08:40.203879 14803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.200:33339
I20240501 20:08:40.290270 16618 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
I20240501 20:08:40.693735 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.714442 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:40.725464 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:40.727113 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.745637 16749 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
I20240501 20:08:40.760160 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.761623 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.770197 15317 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:40.836959 14926 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:41.206208 16879 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
I20240501 20:08:41.352377 14803 ts_manager.cc:292] Set tserver state for 7decc3355b114404abddb7a10cc4358a to MAINTENANCE_MODE
W20240501 20:08:42.769332 16950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:42.769594 16950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:42.777364 16950 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:08:42.972602 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.987289 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.994594 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.023612 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:43.037799 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.085847 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.085848 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.089608 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.097218 14926 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:43.116272 14926 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:43.120503 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.141372 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.155030 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.156275 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:43.262030 15316 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.262495 15317 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.496750 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.514392 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.518571 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:43.547996 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.561158 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.578405 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:08:43.597705 14882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:34095, user_credentials={real_user=slave}} blocked reactor thread for 112942us
I20240501 20:08:43.603461 15317 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.608079 14926 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:43.859109 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.889390 14926 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:43.892544 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:43.924788 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.925807 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.926659 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.935743 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.952737 15317 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:08:44.051024 17073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:44.051432 17073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:08:44.058974 14947 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
W20240501 20:08:44.061816 17073 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:08:44.068389 15338 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Refusing update from remote peer 7decc3355b114404abddb7a10cc4358a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:44.068481 15078 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 7decc3355b114404abddb7a10cc4358a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:44.070262 17003 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:44.071463 17003 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:44.077056 17132 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } }
I20240501 20:08:44.077651 15078 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } }
I20240501 20:08:44.082756 14802 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7decc3355b114404abddb7a10cc4358a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } }
I20240501 20:08:44.086882 15336 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } }
I20240501 20:08:44.128584 14947 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:44.133188 15336 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Refusing update from remote peer 7decc3355b114404abddb7a10cc4358a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:44.133857 17133 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:44.134300 15078 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 7decc3355b114404abddb7a10cc4358a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:44.135085 17133 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:44.139186 15078 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:08:44.140318 15336 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:08:44.137998 17003 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:08:44.145013 14806 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: config changed from index 4 to 5, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "7decc3355b114404abddb7a10cc4358a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:08:44.151571 14790 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c28b52cf6d9d4fa289de88ef97e08bee with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 20:08:44.155115 14880 consensus_peers.cc:485] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a -> Peer 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031): Couldn't send request to peer 98d6b3fd85674334a24cd273bf525324. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c28b52cf6d9d4fa289de88ef97e08bee. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:44.241832 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:44.248245 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:44.257216 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:44.281903 14926 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:44.300954 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:44.303009 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:44.324800 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:44.354727 15317 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:44.510267 17186 ts_tablet_manager.cc:902] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: Initiating tablet copy from peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:44.511919 17186 tablet_copy_client.cc:308] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:44871
I20240501 20:08:44.518111 14967 tablet_copy_service.cc:140] P 7decc3355b114404abddb7a10cc4358a: Received BeginTabletCopySession request for tablet c28b52cf6d9d4fa289de88ef97e08bee from peer 98d6b3fd85674334a24cd273bf525324 ({username='slave'} at 127.14.106.198:36807)
I20240501 20:08:44.518337 14967 tablet_copy_service.cc:161] P 7decc3355b114404abddb7a10cc4358a: Beginning new tablet copy session on tablet c28b52cf6d9d4fa289de88ef97e08bee from peer 98d6b3fd85674334a24cd273bf525324 at {username='slave'} at 127.14.106.198:36807: session id = 98d6b3fd85674334a24cd273bf525324-c28b52cf6d9d4fa289de88ef97e08bee
I20240501 20:08:44.520890 14967 tablet_copy_source_session.cc:205] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:44.525445 17186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b52cf6d9d4fa289de88ef97e08bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:44.538357 17186 tablet_copy_client.cc:783] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:44.538690 17186 tablet_copy_client.cc:647] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:44.541056 17190 tablet_copy_client.cc:1002] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: tablet copy: Injecting latency into file download: 1500
W20240501 20:08:44.851351 17192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:44.851642 17192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:44.859261 17192 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:08:45.000360 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.030900 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.038337 14926 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:45.044307 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:45.063134 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.087793 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.118187 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.166319 15317 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240501 20:08:45.314308 16948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 14865
I20240501 20:08:45.320998 14986 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:45.450492 16948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 14865
W20240501 20:08:45.521504 15271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:45.523128 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:45.587319 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.602129 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.604039 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.619093 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:45.623189 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.625553 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.625609 15317 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:08:45.645044 15011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:45.677371 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:08:45.732697 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:08:45.760299 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:08:45.785081 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:45.831028 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.841749 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.850020 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.851641 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:45.856818 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:45.868829 15317 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.876015 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:08:45.876581 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:08:45.907517 15141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:45.915448 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:08:45.945797 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:45.973917 15206 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:08:45.977344 15078 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:45.977952 17349 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:45.978055 15336 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:45.978665 17038 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:08:45.986869 15143 consensus_peers.cc:485] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d -> Peer 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197:35831): Couldn't send request to peer 250badc514c94bcdbe2ace1b62a0874f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efbf0e49f78a4ff099786f47eedc3c8b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:45.992597 17038 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:08:45.996475 15336 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:08:45.999491 15078 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:08:45.999979 14806 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d reported cstate change: config changed from index -1 to 4, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:46.021415 15206 consensus_queue.cc:237] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: NON_VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: true } }
I20240501 20:08:46.024112 15336 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:46.024530 15078 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:46.025068 17352 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:46.025440 17349 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:46.028663 17038 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: NON_VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: true } } }
W20240501 20:08:46.031590 15143 consensus_peers.cc:485] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d -> Peer 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199:38431): Couldn't send request to peer 8dddd9d9b8fb47cb944ad4179152a915. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07797a04ba4b4ee3801855bfdbc1720d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:46.031989 15078 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: NON_VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: true } } }
I20240501 20:08:46.039372 14805 catalog_manager.cc:5484] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: NON_VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: true } } }
I20240501 20:08:46.041708 17186 tablet_copy_client.cc:523] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:46.045181 15338 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: NON_VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: true } } }
I20240501 20:08:46.048781 17186 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: Bootstrap starting.
I20240501 20:08:46.062747 17186 log.cc:824] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:46.074669 15338 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:08:46.077523 15078 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:46.077719 15208 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:46.078068 17365 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:46.078826 17366 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:46.083330 16947 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
I20240501 20:08:46.083859 15078 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
W20240501 20:08:46.085453 15271 consensus_peers.cc:485] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 -> Peer 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973): Couldn't send request to peer 0d9d84f0f3224d1a8fbaedd1021d730f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e093bed4aae0484faa3cba7a4cc52ff9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:46.085469 15208 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
I20240501 20:08:46.089001 14797 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 reported cstate change: config changed from index -1 to 4, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:46.133275 17186 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:46.134033 17186 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: Bootstrap complete.
I20240501 20:08:46.134727 17186 ts_tablet_manager.cc:1367] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: Time spent bootstrapping tablet: real 0.086s user 0.057s sys 0.011s
I20240501 20:08:46.141752 17186 raft_consensus.cc:348] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:46.142282 17186 raft_consensus.cc:729] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98d6b3fd85674334a24cd273bf525324, State: Initialized, Role: LEARNER
I20240501 20:08:46.142935 17186 consensus_queue.cc:260] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:46.144923 17186 ts_tablet_manager.cc:1398] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20240501 20:08:46.146464 17186 tablet_copy_client.cc:1108] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.530401 17410 ts_tablet_manager.cc:902] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: Initiating tablet copy from peer 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095)
I20240501 20:08:46.532433 17410 tablet_copy_client.cc:308] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:34095
I20240501 20:08:46.540760 15228 tablet_copy_service.cc:140] P 2177fb70b1324acb81b2bea7be19955d: Received BeginTabletCopySession request for tablet efbf0e49f78a4ff099786f47eedc3c8b from peer 250badc514c94bcdbe2ace1b62a0874f ({username='slave'} at 127.14.106.197:38403)
I20240501 20:08:46.541070 15228 tablet_copy_service.cc:161] P 2177fb70b1324acb81b2bea7be19955d: Beginning new tablet copy session on tablet efbf0e49f78a4ff099786f47eedc3c8b from peer 250badc514c94bcdbe2ace1b62a0874f at {username='slave'} at 127.14.106.197:38403: session id = 250badc514c94bcdbe2ace1b62a0874f-efbf0e49f78a4ff099786f47eedc3c8b
I20240501 20:08:46.545491 15228 tablet_copy_source_session.cc:205] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:46.546978 17416 ts_tablet_manager.cc:902] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: Initiating tablet copy from peer 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095)
I20240501 20:08:46.550240 17410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbf0e49f78a4ff099786f47eedc3c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:46.551856 17416 tablet_copy_client.cc:308] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:34095
I20240501 20:08:46.565107 15228 tablet_copy_service.cc:140] P 2177fb70b1324acb81b2bea7be19955d: Received BeginTabletCopySession request for tablet 07797a04ba4b4ee3801855bfdbc1720d from peer 8dddd9d9b8fb47cb944ad4179152a915 ({username='slave'} at 127.14.106.199:36139)
I20240501 20:08:46.565292 15228 tablet_copy_service.cc:161] P 2177fb70b1324acb81b2bea7be19955d: Beginning new tablet copy session on tablet 07797a04ba4b4ee3801855bfdbc1720d from peer 8dddd9d9b8fb47cb944ad4179152a915 at {username='slave'} at 127.14.106.199:36139: session id = 8dddd9d9b8fb47cb944ad4179152a915-07797a04ba4b4ee3801855bfdbc1720d
I20240501 20:08:46.565857 17410 tablet_copy_client.cc:783] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:46.566246 17410 tablet_copy_client.cc:647] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:46.567720 15228 tablet_copy_source_session.cc:205] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:46.571739 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07797a04ba4b4ee3801855bfdbc1720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:46.578732 17038 raft_consensus.cc:482] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.578907 17038 raft_consensus.cc:504] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:46.579710 17038 leader_election.cc:288] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:46.580065 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e54c2e99947456988c55a7e5c5c4f53" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:46.580538 17365 raft_consensus.cc:482] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.580713 17365 raft_consensus.cc:504] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:46.581300 17365 leader_election.cc:288] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:08:46.581900 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" is_pre_election: true
W20240501 20:08:46.582895 15271 leader_election.cc:334] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.582979 17416 tablet_copy_client.cc:783] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:46.583048 15271 leader_election.cc:302] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e494ca2918594ba792c2d37d98b8c607; no voters: 4ed2ac58f951430a8c7e084f97d7a2a4, 7decc3355b114404abddb7a10cc4358a
I20240501 20:08:46.583336 17416 tablet_copy_client.cc:647] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:46.583354 17365 raft_consensus.cc:2726] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:08:46.588038 17414 tablet_copy_client.cc:1002] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Injecting latency into file download: 1500
I20240501 20:08:46.588119 17420 tablet_copy_client.cc:1002] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: tablet copy: Injecting latency into file download: 1500
W20240501 20:08:46.588734 15141 leader_election.cc:334] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.588850 15141 leader_election.cc:302] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d; no voters: 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607
I20240501 20:08:46.589156 17038 raft_consensus.cc:2726] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:08:46.628706 17437 ts_tablet_manager.cc:902] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: Initiating tablet copy from peer e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:46.630400 17437 tablet_copy_client.cc:308] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:39333
I20240501 20:08:46.637244 15358 tablet_copy_service.cc:140] P e494ca2918594ba792c2d37d98b8c607: Received BeginTabletCopySession request for tablet e093bed4aae0484faa3cba7a4cc52ff9 from peer 0d9d84f0f3224d1a8fbaedd1021d730f ({username='slave'} at 127.14.106.200:39839)
I20240501 20:08:46.637439 15358 tablet_copy_service.cc:161] P e494ca2918594ba792c2d37d98b8c607: Beginning new tablet copy session on tablet e093bed4aae0484faa3cba7a4cc52ff9 from peer 0d9d84f0f3224d1a8fbaedd1021d730f at {username='slave'} at 127.14.106.200:39839: session id = 0d9d84f0f3224d1a8fbaedd1021d730f-e093bed4aae0484faa3cba7a4cc52ff9
I20240501 20:08:46.640511 15358 tablet_copy_source_session.cc:205] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:46.645186 17437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e093bed4aae0484faa3cba7a4cc52ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:46.655465 17437 tablet_copy_client.cc:783] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:46.655777 17437 tablet_copy_client.cc:647] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:46.657876 17441 tablet_copy_client.cc:1002] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Injecting latency into file download: 1500
I20240501 20:08:46.676050 17038 raft_consensus.cc:482] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.676270 17038 raft_consensus.cc:504] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:46.676949 17038 leader_election.cc:288] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:46.677496 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e873682369f4d4f93dbe5fb3b369d1f" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
W20240501 20:08:46.678617 15141 leader_election.cc:334] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.678776 15141 leader_election.cc:302] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d; no voters: 7decc3355b114404abddb7a10cc4358a, e494ca2918594ba792c2d37d98b8c607
I20240501 20:08:46.679039 17038 raft_consensus.cc:2726] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:08:46.700440 17365 raft_consensus.cc:482] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.700685 17365 raft_consensus.cc:504] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:46.701339 17365 leader_election.cc:288] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095)
I20240501 20:08:46.701951 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e65bcdb5174beb91a9edb54d20a102" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
W20240501 20:08:46.703070 15271 leader_election.cc:334] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.703184 15271 leader_election.cc:302] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e494ca2918594ba792c2d37d98b8c607; no voters: 2177fb70b1324acb81b2bea7be19955d, 7decc3355b114404abddb7a10cc4358a
I20240501 20:08:46.703464 17365 raft_consensus.cc:2726] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:08:46.758080 17385 raft_consensus.cc:482] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.758329 17385 raft_consensus.cc:504] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:46.759107 17385 leader_election.cc:288] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:46.760129 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:46.760372 15338 raft_consensus.cc:2454] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 1.
W20240501 20:08:46.760525 15011 leader_election.cc:334] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.760774 15012 leader_election.cc:302] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, e494ca2918594ba792c2d37d98b8c607; no voters: 7decc3355b114404abddb7a10cc4358a
I20240501 20:08:46.761080 17385 raft_consensus.cc:2781] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:08:46.761204 17385 raft_consensus.cc:482] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.761312 17385 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:46.787915 17038 raft_consensus.cc:482] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.788152 17038 raft_consensus.cc:504] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:46.788763 17038 leader_election.cc:288] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:46.789436 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e65bcdb5174beb91a9edb54d20a102" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607" is_pre_election: true
I20240501 20:08:46.789686 15338 raft_consensus.cc:2454] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 1.
I20240501 20:08:46.790122 15142 leader_election.cc:302] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters:
W20240501 20:08:46.790306 15141 leader_election.cc:334] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.790402 17038 raft_consensus.cc:2781] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:08:46.790524 17038 raft_consensus.cc:482] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.790657 17038 raft_consensus.cc:3037] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:46.806066 17365 raft_consensus.cc:482] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.806257 17365 raft_consensus.cc:504] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:46.807058 17365 leader_election.cc:288] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:46.807606 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e54c2e99947456988c55a7e5c5c4f53" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:46.807843 15208 raft_consensus.cc:2454] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 1.
I20240501 20:08:46.808274 15273 leader_election.cc:302] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:46.808560 17365 raft_consensus.cc:2781] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 20:08:46.808583 15271 leader_election.cc:334] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.808684 17365 raft_consensus.cc:482] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.808825 17365 raft_consensus.cc:3037] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:46.820868 17038 raft_consensus.cc:504] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:46.820904 17385 raft_consensus.cc:504] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:46.821601 17385 leader_election.cc:288] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 2 election: Requested vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:46.821635 17038 leader_election.cc:288] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 election: Requested vote from peers 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:46.822340 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e65bcdb5174beb91a9edb54d20a102" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:46.822561 15338 raft_consensus.cc:3037] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:46.822533 17365 raft_consensus.cc:504] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:46.823190 15336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:46.823391 15336 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:46.823235 17365 leader_election.cc:288] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
W20240501 20:08:46.827180 15011 leader_election.cc:334] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.827257 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e54c2e99947456988c55a7e5c5c4f53" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:46.827450 15208 raft_consensus.cc:3037] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:08:46.828899 15271 leader_election.cc:334] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:46.835265 15141 leader_election.cc:334] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.918078 15338 raft_consensus.cc:2454] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 2.
I20240501 20:08:46.918051 15208 raft_consensus.cc:2454] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 2.
I20240501 20:08:46.918224 15336 raft_consensus.cc:2454] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed2ac58f951430a8c7e084f97d7a2a4 in term 2.
I20240501 20:08:46.918709 15273 leader_election.cc:302] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters: 7decc3355b114404abddb7a10cc4358a
I20240501 20:08:46.918699 15142 leader_election.cc:302] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters: 7decc3355b114404abddb7a10cc4358a
I20240501 20:08:46.919029 15012 leader_election.cc:302] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ed2ac58f951430a8c7e084f97d7a2a4, e494ca2918594ba792c2d37d98b8c607; no voters: 7decc3355b114404abddb7a10cc4358a
I20240501 20:08:46.919171 17365 raft_consensus.cc:2781] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:46.919215 17038 raft_consensus.cc:2781] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:46.919809 17038 raft_consensus.cc:686] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Becoming Leader. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Running, Role: LEADER
I20240501 20:08:46.920190 17385 raft_consensus.cc:2781] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:46.920325 17385 raft_consensus.cc:686] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Becoming Leader. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Running, Role: LEADER
I20240501 20:08:46.920125 17038 consensus_queue.cc:237] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:08:46.920745 17385 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:46.925182 17365 raft_consensus.cc:686] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Becoming Leader. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Running, Role: LEADER
I20240501 20:08:46.925618 14805 catalog_manager.cc:5484] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d reported cstate change: term changed from 1 to 2, leader changed from 7decc3355b114404abddb7a10cc4358a (127.14.106.193) to 2177fb70b1324acb81b2bea7be19955d (127.14.106.195). New cstate: current_term: 2 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:46.925810 17365 consensus_queue.cc:237] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:46.926445 14806 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: term changed from 1 to 2, leader changed from 7decc3355b114404abddb7a10cc4358a (127.14.106.193) to 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194). New cstate: current_term: 2 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:46.928603 14797 catalog_manager.cc:5484] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 reported cstate change: term changed from 1 to 2, leader changed from 7decc3355b114404abddb7a10cc4358a (127.14.106.193) to e494ca2918594ba792c2d37d98b8c607 (127.14.106.196). New cstate: current_term: 2 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:46.950724 17365 raft_consensus.cc:482] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.950933 17365 raft_consensus.cc:504] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:46.951552 17365 leader_election.cc:288] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:46.951977 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e873682369f4d4f93dbe5fb3b369d1f" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d" is_pre_election: true
I20240501 20:08:46.952247 15208 raft_consensus.cc:2454] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 1.
I20240501 20:08:46.952817 15273 leader_election.cc:302] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters:
W20240501 20:08:46.953068 15271 leader_election.cc:334] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:46.953255 17365 raft_consensus.cc:2781] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:08:46.953380 17365 raft_consensus.cc:482] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7decc3355b114404abddb7a10cc4358a)
I20240501 20:08:46.953496 17365 raft_consensus.cc:3037] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:47.014976 17365 raft_consensus.cc:504] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:47.015630 17365 leader_election.cc:288] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:08:47.016125 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e873682369f4d4f93dbe5fb3b369d1f" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:47.016391 15208 raft_consensus.cc:3037] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:08:47.017055 15271 leader_election.cc:334] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:47.019028 15208 raft_consensus.cc:2454] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 2.
I20240501 20:08:47.019441 15273 leader_election.cc:302] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters: 7decc3355b114404abddb7a10cc4358a
I20240501 20:08:47.019738 17458 raft_consensus.cc:2781] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:47.019927 17458 raft_consensus.cc:686] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Becoming Leader. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Running, Role: LEADER
I20240501 20:08:47.020248 17458 consensus_queue.cc:237] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:08:47.023236 14797 catalog_manager.cc:5484] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 reported cstate change: term changed from 1 to 2, leader changed from 7decc3355b114404abddb7a10cc4358a (127.14.106.193) to e494ca2918594ba792c2d37d98b8c607 (127.14.106.196). New cstate: current_term: 2 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:47.377203 15336 raft_consensus.cc:1261] T 95e65bcdb5174beb91a9edb54d20a102 P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:08:47.377797 17467 consensus_queue.cc:1035] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:08:47.383036 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:47.390767 15336 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:08:47.391330 17385 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:08:47.397631 15011 consensus_peers.cc:485] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:08:47.400803 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:47.405510 16573 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 1 LEARNER]: Advancing to term 2
I20240501 20:08:47.418920 15208 raft_consensus.cc:1261] T 3e873682369f4d4f93dbe5fb3b369d1f P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:08:47.419533 17458 consensus_queue.cc:1035] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:47.428100 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 LEARNER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:08:47.428860 17505 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:08:47.433585 17505 raft_consensus.cc:1050] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: attempting to promote NON_VOTER 98d6b3fd85674334a24cd273bf525324 to VOTER
I20240501 20:08:47.434835 17505 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:08:47.437330 15336 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240501 20:08:47.438701 15011 consensus_peers.cc:485] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:47.439137 17394 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:08:47.440119 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 LEARNER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:08:47.440860 17505 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:08:47.443092 17512 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:47.443717 16573 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
W20240501 20:08:47.516943 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:47.533411 14797 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 reported cstate change: config changed from index 5 to 7, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:47.537727 15336 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:47.546278 15078 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:08:47.548276 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:08:47.548844 17394 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:47.549392 15208 raft_consensus.cc:1261] T 7e54c2e99947456988c55a7e5c5c4f53 P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:08:47.549988 17458 consensus_queue.cc:1035] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:47.551604 17505 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7decc3355b114404abddb7a10cc4358a (127.14.106.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:47.552191 16573 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7decc3355b114404abddb7a10cc4358a (127.14.106.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:47.633035 14793 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c28b52cf6d9d4fa289de88ef97e08bee with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:47.633195 15338 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:08:47.634318 14806 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 reported cstate change: config changed from index 7 to 8, VOTER 7decc3355b114404abddb7a10cc4358a (127.14.106.193) evicted. New cstate: current_term: 2 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:47.635063 17394 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:47.636682 15338 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7decc3355b114404abddb7a10cc4358a (127.14.106.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
W20240501 20:08:47.642092 14790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:47.643635 14790 catalog_manager.cc:4573] TS 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c28b52cf6d9d4fa289de88ef97e08bee: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:47.906677 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:08:47.924132 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:08:48.088655 17410 tablet_copy_client.cc:523] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:48.088732 17416 tablet_copy_client.cc:523] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:48.147859 17416 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: Bootstrap starting.
I20240501 20:08:48.147887 17410 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap starting.
I20240501 20:08:48.156919 17416 log.cc:824] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:48.157299 17410 log.cc:824] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:48.158407 17437 tablet_copy_client.cc:523] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:48.207158 17410 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:48.207244 17416 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:48.207803 17416 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: Bootstrap complete.
I20240501 20:08:48.207803 17410 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap complete.
I20240501 20:08:48.208307 17410 ts_tablet_manager.cc:1367] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: Time spent bootstrapping tablet: real 0.061s user 0.060s sys 0.000s
I20240501 20:08:48.208307 17416 ts_tablet_manager.cc:1367] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: Time spent bootstrapping tablet: real 0.061s user 0.049s sys 0.011s
I20240501 20:08:48.215116 17410 raft_consensus.cc:348] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:08:48.215116 17416 raft_consensus.cc:348] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: NON_VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: true } }
I20240501 20:08:48.215646 17416 raft_consensus.cc:729] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8dddd9d9b8fb47cb944ad4179152a915, State: Initialized, Role: LEARNER
I20240501 20:08:48.215651 17410 raft_consensus.cc:729] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 250badc514c94bcdbe2ace1b62a0874f, State: Initialized, Role: LEARNER
I20240501 20:08:48.216228 17410 consensus_queue.cc:260] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:08:48.216228 17416 consensus_queue.cc:260] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: NON_VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: true } }
I20240501 20:08:48.218956 17410 ts_tablet_manager.cc:1398] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20240501 20:08:48.220441 17416 ts_tablet_manager.cc:1398] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20240501 20:08:48.221693 15228 tablet_copy_service.cc:342] P 2177fb70b1324acb81b2bea7be19955d: Request end of tablet copy session 250badc514c94bcdbe2ace1b62a0874f-efbf0e49f78a4ff099786f47eedc3c8b received from {username='slave'} at 127.14.106.197:38403
I20240501 20:08:48.221853 15228 tablet_copy_service.cc:433] P 2177fb70b1324acb81b2bea7be19955d: ending tablet copy session 250badc514c94bcdbe2ace1b62a0874f-efbf0e49f78a4ff099786f47eedc3c8b on tablet efbf0e49f78a4ff099786f47eedc3c8b with peer 250badc514c94bcdbe2ace1b62a0874f
I20240501 20:08:48.222388 15227 tablet_copy_service.cc:342] P 2177fb70b1324acb81b2bea7be19955d: Request end of tablet copy session 8dddd9d9b8fb47cb944ad4179152a915-07797a04ba4b4ee3801855bfdbc1720d received from {username='slave'} at 127.14.106.199:36139
I20240501 20:08:48.222506 15227 tablet_copy_service.cc:433] P 2177fb70b1324acb81b2bea7be19955d: ending tablet copy session 8dddd9d9b8fb47cb944ad4179152a915-07797a04ba4b4ee3801855bfdbc1720d on tablet 07797a04ba4b4ee3801855bfdbc1720d with peer 8dddd9d9b8fb47cb944ad4179152a915
I20240501 20:08:48.236274 16704 raft_consensus.cc:1203] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 20:08:48.241901 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:08:48.246672 17437 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap starting.
I20240501 20:08:48.255790 17437 log.cc:824] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: Log is configured to *not* fsync() on all Append() calls
W20240501 20:08:48.279089 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:08:48.284874 16442 raft_consensus.cc:1203] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:48.294512 17437 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:48.295048 17437 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap complete.
I20240501 20:08:48.295471 17437 ts_tablet_manager.cc:1367] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: Time spent bootstrapping tablet: real 0.049s user 0.038s sys 0.011s
I20240501 20:08:48.299953 17437 raft_consensus.cc:348] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:08:48.300338 17437 raft_consensus.cc:729] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d9d84f0f3224d1a8fbaedd1021d730f, State: Initialized, Role: LEARNER
I20240501 20:08:48.300812 17437 consensus_queue.cc:260] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:08:48.302464 17437 ts_tablet_manager.cc:1398] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240501 20:08:48.303493 15358 tablet_copy_service.cc:342] P e494ca2918594ba792c2d37d98b8c607: Request end of tablet copy session 0d9d84f0f3224d1a8fbaedd1021d730f-e093bed4aae0484faa3cba7a4cc52ff9 received from {username='slave'} at 127.14.106.200:39839
I20240501 20:08:48.303650 15358 tablet_copy_service.cc:433] P e494ca2918594ba792c2d37d98b8c607: ending tablet copy session 0d9d84f0f3224d1a8fbaedd1021d730f-e093bed4aae0484faa3cba7a4cc52ff9 on tablet e093bed4aae0484faa3cba7a4cc52ff9 with peer 0d9d84f0f3224d1a8fbaedd1021d730f
W20240501 20:08:48.383939 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:08:48.431936 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:08:48.475248 16834 raft_consensus.cc:1203] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 20:08:48.513900 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:08:48.656004 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:08:48.762091 17616 raft_consensus.cc:1050] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: attempting to promote NON_VOTER 250badc514c94bcdbe2ace1b62a0874f to VOTER
I20240501 20:08:48.763273 17616 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:08:48.766503 15338 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.766574 15078 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.766877 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 1 LEARNER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.767153 17615 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:48.767503 17616 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:48.767896 17038 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:48.773301 17038 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:48.774271 16442 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:48.776983 15078 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:48.817230 17621 raft_consensus.cc:1050] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: attempting to promote NON_VOTER 8dddd9d9b8fb47cb944ad4179152a915 to VOTER
I20240501 20:08:48.818099 17621 consensus_queue.cc:237] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } }
I20240501 20:08:48.820572 16704 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 LEARNER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.820629 15338 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.820725 15077 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.821377 17615 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:48.821807 17621 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:48.822302 17623 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:48.825399 17621 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } }
I20240501 20:08:48.826045 15077 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } }
I20240501 20:08:48.826921 16704 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } }
I20240501 20:08:48.864114 14797 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: config changed from index 4 to 5, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:48.866319 15338 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:48.866367 15336 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } }
W20240501 20:08:48.876765 14790 catalog_manager.cc:4573] TS 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c28b52cf6d9d4fa289de88ef97e08bee: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
I20240501 20:08:48.887554 14806 catalog_manager.cc:5484] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d reported cstate change: config changed from index 4 to 5, 8dddd9d9b8fb47cb944ad4179152a915 (127.14.106.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:08:48.892125 15208 consensus_queue.cc:237] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } }
I20240501 20:08:48.893818 16704 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:08:48.893869 15077 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:08:48.894541 17615 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:08:48.895680 17038 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:08:48.897722 17615 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } }
I20240501 20:08:48.899705 16703 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } }
I20240501 20:08:48.900883 15077 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } }
I20240501 20:08:48.930976 15208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b"
dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41906
I20240501 20:08:48.931241 15208 raft_consensus.cc:595] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:48.962332 14792 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 07797a04ba4b4ee3801855bfdbc1720d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:48.962801 14797 catalog_manager.cc:5484] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 reported cstate change: config changed from index 5 to 6, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } }
I20240501 20:08:48.963138 17621 raft_consensus.cc:979] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: : Instructing follower e494ca2918594ba792c2d37d98b8c607 to start an election
I20240501 20:08:48.963279 17621 raft_consensus.cc:1067] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Signalling peer e494ca2918594ba792c2d37d98b8c607 to start an election
I20240501 20:08:48.964321 15338 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b"
dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
from {username='slave'} at 127.14.106.195:41589
I20240501 20:08:48.964586 15338 raft_consensus.cc:482] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:48.964716 15338 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:48.968127 15317 tablet_service.cc:1514] Processing DeleteTablet for tablet 07797a04ba4b4ee3801855bfdbc1720d with delete_type TABLET_DATA_TOMBSTONED (TS e494ca2918594ba792c2d37d98b8c607 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44314
I20240501 20:08:48.972733 17656 tablet_replica.cc:330] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: stopping tablet replica
I20240501 20:08:48.973361 17656 raft_consensus.cc:2229] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:48.973834 17656 raft_consensus.cc:2258] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:48.975862 17656 ts_tablet_manager.cc:1875] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:48.978343 17469 raft_consensus.cc:1050] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: attempting to promote NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f to VOTER
I20240501 20:08:48.979403 17469 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:08:48.981913 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 LEARNER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.982686 15077 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.982712 15208 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:48.983397 17659 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:48.983722 17655 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:48.983943 17469 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:49.002157 17659 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:49.006680 15206 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:49.006481 16834 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:49.062669 15338 raft_consensus.cc:504] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:08:49.064491 15338 leader_election.cc:288] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 election: Requested vote from peers 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059), 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197:35831)
I20240501 20:08:49.065454 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:49.065681 15077 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:49.068470 15077 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 2.
I20240501 20:08:49.069741 17656 ts_tablet_manager.cc:1888] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:08:49.070022 17656 log.cc:1196] T 07797a04ba4b4ee3801855bfdbc1720d P e494ca2918594ba792c2d37d98b8c607: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/07797a04ba4b4ee3801855bfdbc1720d
I20240501 20:08:49.070004 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:08:49.070264 15206 raft_consensus.cc:3032] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:49.070380 15206 raft_consensus.cc:729] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Running, Role: LEADER
I20240501 20:08:49.071455 14791 catalog_manager.cc:4830] TS e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333): tablet 07797a04ba4b4ee3801855bfdbc1720d (table rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]) successfully deleted
I20240501 20:08:49.071931 14803 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d reported cstate change: config changed from index 4 to 5, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:49.073295 15206 consensus_queue.cc:260] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:08:49.073822 15206 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:49.076649 15317 tablet_service.cc:1514] Processing DeleteTablet for tablet 07797a04ba4b4ee3801855bfdbc1720d with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:44314
I20240501 20:08:49.077178 14791 catalog_manager.cc:4830] TS e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333): tablet 07797a04ba4b4ee3801855bfdbc1720d (table rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]) successfully deleted
I20240501 20:08:49.079651 16442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "e494ca2918594ba792c2d37d98b8c607" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "250badc514c94bcdbe2ace1b62a0874f"
I20240501 20:08:49.079914 16442 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:49.167794 16442 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 2.
I20240501 20:08:49.167795 15206 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e494ca2918594ba792c2d37d98b8c607 in term 2.
I20240501 20:08:49.168710 15273 leader_election.cc:302] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 4ed2ac58f951430a8c7e084f97d7a2a4, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:49.169092 17670 raft_consensus.cc:2781] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:49.169389 17670 raft_consensus.cc:686] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Becoming Leader. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Running, Role: LEADER
I20240501 20:08:49.169816 17670 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:08:49.169956 15077 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:49.174600 15336 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:08:49.176350 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:08:49.176893 17670 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:08:49.177045 15206 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:08:49.177623 17670 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:08:49.179147 14805 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 reported cstate change: term changed from 1 to 2, leader changed from 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) to e494ca2918594ba792c2d37d98b8c607 (127.14.106.196). New cstate: current_term: 2 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:49.179453 17670 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:49.181082 15206 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:49.183256 16834 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:49.278158 14791 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e093bed4aae0484faa3cba7a4cc52ff9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:49.278931 14797 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 reported cstate change: config changed from index 5 to 6, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:08:49.282747 15057 tablet_service.cc:1514] Processing DeleteTablet for tablet e093bed4aae0484faa3cba7a4cc52ff9 with delete_type TABLET_DATA_TOMBSTONED (TS 4ed2ac58f951430a8c7e084f97d7a2a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60194
I20240501 20:08:49.283754 17676 tablet_replica.cc:330] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: stopping tablet replica
I20240501 20:08:49.284150 17676 raft_consensus.cc:2229] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:49.284482 17676 raft_consensus.cc:2258] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:49.285810 17676 ts_tablet_manager.cc:1875] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:49.373566 17676 ts_tablet_manager.cc:1888] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:08:49.373976 17676 log.cc:1196] T e093bed4aae0484faa3cba7a4cc52ff9 P 4ed2ac58f951430a8c7e084f97d7a2a4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/e093bed4aae0484faa3cba7a4cc52ff9
I20240501 20:08:49.375588 14793 catalog_manager.cc:4830] TS 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059): tablet e093bed4aae0484faa3cba7a4cc52ff9 (table rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]) successfully deleted
I20240501 20:08:49.588461 15206 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:08:49.589051 17670 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:08:49.675968 15077 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:08:49.676581 17670 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:08:49.687844 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:08:49.688534 17469 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:08:49.946143 14790 catalog_manager.cc:4573] TS 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c28b52cf6d9d4fa289de88ef97e08bee: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:50.071061 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:08:50.095839 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:08:50.120010 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:08:50.242709 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:08:50.266510 15336 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:08:50.268195 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:08:50.268260 15077 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:08:50.268682 17721 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:08:50.268996 17720 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:08:50.271098 17469 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:50.271762 15077 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:50.274793 16442 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:50.287772 14791 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efbf0e49f78a4ff099786f47eedc3c8b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:50.288973 14805 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 reported cstate change: config changed from index 5 to 7, VOTER 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) evicted. New cstate: current_term: 2 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:08:50.295193 15188 tablet_service.cc:1514] Processing DeleteTablet for tablet efbf0e49f78a4ff099786f47eedc3c8b with delete_type TABLET_DATA_TOMBSTONED (TS 2177fb70b1324acb81b2bea7be19955d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34738
I20240501 20:08:50.296049 17727 tablet_replica.cc:330] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: stopping tablet replica
I20240501 20:08:50.296430 17727 raft_consensus.cc:2229] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:50.296658 17727 raft_consensus.cc:2258] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:50.297885 17727 ts_tablet_manager.cc:1875] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:08:50.382285 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:08:50.490396 17727 ts_tablet_manager.cc:1888] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:08:50.490607 17727 log.cc:1196] T efbf0e49f78a4ff099786f47eedc3c8b P 2177fb70b1324acb81b2bea7be19955d: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/efbf0e49f78a4ff099786f47eedc3c8b
I20240501 20:08:50.491601 14792 catalog_manager.cc:4830] TS 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095): tablet efbf0e49f78a4ff099786f47eedc3c8b (table rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]) successfully deleted
W20240501 20:08:50.562140 15271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 20:08:50.589613 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:50.591077 16814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:50.591507 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:50.611343 15057 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:50.615134 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:50.616885 15317 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:50.646996 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:08:50.652137 15011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240501 20:08:50.654042 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:08:50.707038 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:08:50.723193 15206 consensus_queue.cc:237] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:08:50.725827 16703 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:08:50.726183 15143 consensus_peers.cc:485] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d -> Peer 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197:35831): Couldn't send request to peer 250badc514c94bcdbe2ace1b62a0874f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07797a04ba4b4ee3801855bfdbc1720d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:50.726699 15077 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:08:50.726678 17038 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:08:50.727319 17038 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:08:50.730257 17777 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:08:50.731331 15077 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:08:50.739185 16703 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
W20240501 20:08:50.787528 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:08:50.799883 14806 catalog_manager.cc:5484] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: config changed from index 6 to 7, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:08:50.817499 15336 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:08:50.819674 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:08:50.819948 15077 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:08:50.820417 17790 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:50.820725 17720 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:08:50.821009 15271 consensus_peers.cc:485] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 -> Peer 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973): Couldn't send request to peer 0d9d84f0f3224d1a8fbaedd1021d730f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efbf0e49f78a4ff099786f47eedc3c8b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:50.823137 17790 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
I20240501 20:08:50.823585 15077 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
I20240501 20:08:50.827050 16442 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
W20240501 20:08:50.846980 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:08:50.902915 14797 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 reported cstate change: config changed from index 7 to 8, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New cstate: current_term: 2 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:08:50.942824 15141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:08:50.962194 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:08:51.014550 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:08:51.163568 17813 ts_tablet_manager.cc:902] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: Initiating tablet copy from peer 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095)
I20240501 20:08:51.164803 17813 tablet_copy_client.cc:308] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:34095
I20240501 20:08:51.165432 15227 tablet_copy_service.cc:140] P 2177fb70b1324acb81b2bea7be19955d: Received BeginTabletCopySession request for tablet 07797a04ba4b4ee3801855bfdbc1720d from peer 250badc514c94bcdbe2ace1b62a0874f ({username='slave'} at 127.14.106.197:38403)
I20240501 20:08:51.165565 15227 tablet_copy_service.cc:161] P 2177fb70b1324acb81b2bea7be19955d: Beginning new tablet copy session on tablet 07797a04ba4b4ee3801855bfdbc1720d from peer 250badc514c94bcdbe2ace1b62a0874f at {username='slave'} at 127.14.106.197:38403: session id = 250badc514c94bcdbe2ace1b62a0874f-07797a04ba4b4ee3801855bfdbc1720d
I20240501 20:08:51.168246 15227 tablet_copy_source_session.cc:205] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:51.169390 17813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07797a04ba4b4ee3801855bfdbc1720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:51.173946 17813 tablet_copy_client.cc:783] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:51.174180 17813 tablet_copy_client.cc:647] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:51.175664 17817 tablet_copy_client.cc:1002] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Injecting latency into file download: 1500
I20240501 20:08:51.215308 17818 ts_tablet_manager.cc:902] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: Initiating tablet copy from peer e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:51.216696 17818 tablet_copy_client.cc:308] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:39333
I20240501 20:08:51.217330 15358 tablet_copy_service.cc:140] P e494ca2918594ba792c2d37d98b8c607: Received BeginTabletCopySession request for tablet efbf0e49f78a4ff099786f47eedc3c8b from peer 0d9d84f0f3224d1a8fbaedd1021d730f ({username='slave'} at 127.14.106.200:39839)
I20240501 20:08:51.217499 15358 tablet_copy_service.cc:161] P e494ca2918594ba792c2d37d98b8c607: Beginning new tablet copy session on tablet efbf0e49f78a4ff099786f47eedc3c8b from peer 0d9d84f0f3224d1a8fbaedd1021d730f at {username='slave'} at 127.14.106.200:39839: session id = 0d9d84f0f3224d1a8fbaedd1021d730f-efbf0e49f78a4ff099786f47eedc3c8b
I20240501 20:08:51.219898 15358 tablet_copy_source_session.cc:205] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:51.221140 17818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbf0e49f78a4ff099786f47eedc3c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:51.225680 17818 tablet_copy_client.cc:783] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:51.225977 17818 tablet_copy_client.cc:647] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:51.227530 17822 tablet_copy_client.cc:1002] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Injecting latency into file download: 1500
W20240501 20:08:52.045225 14790 catalog_manager.cc:4573] TS 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c28b52cf6d9d4fa289de88ef97e08bee: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:52.303920 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:08:52.445183 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:08:52.524247 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:08:52.676020 17813 tablet_copy_client.cc:523] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:52.678968 17813 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap starting.
I20240501 20:08:52.708459 17813 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:52.708882 17813 tablet_bootstrap.cc:492] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap complete.
I20240501 20:08:52.709125 17813 ts_tablet_manager.cc:1367] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.012s
I20240501 20:08:52.709903 17813 raft_consensus.cc:348] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:08:52.710117 17813 raft_consensus.cc:729] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 250badc514c94bcdbe2ace1b62a0874f, State: Initialized, Role: LEARNER
I20240501 20:08:52.710311 17813 consensus_queue.cc:260] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:08:52.711126 17813 ts_tablet_manager.cc:1398] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 20:08:52.712733 15227 tablet_copy_service.cc:342] P 2177fb70b1324acb81b2bea7be19955d: Request end of tablet copy session 250badc514c94bcdbe2ace1b62a0874f-07797a04ba4b4ee3801855bfdbc1720d received from {username='slave'} at 127.14.106.197:38403
I20240501 20:08:52.712863 15227 tablet_copy_service.cc:433] P 2177fb70b1324acb81b2bea7be19955d: ending tablet copy session 250badc514c94bcdbe2ace1b62a0874f-07797a04ba4b4ee3801855bfdbc1720d on tablet 07797a04ba4b4ee3801855bfdbc1720d with peer 250badc514c94bcdbe2ace1b62a0874f
I20240501 20:08:52.727895 17818 tablet_copy_client.cc:523] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:52.730473 17818 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap starting.
I20240501 20:08:52.763060 17818 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:52.763463 17818 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap complete.
I20240501 20:08:52.763710 17818 ts_tablet_manager.cc:1367] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.004s
I20240501 20:08:52.764492 17818 raft_consensus.cc:348] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:08:52.764706 17818 raft_consensus.cc:729] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0d9d84f0f3224d1a8fbaedd1021d730f, State: Initialized, Role: LEARNER
I20240501 20:08:52.764901 17818 consensus_queue.cc:260] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:08:52.765547 17818 ts_tablet_manager.cc:1398] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:52.766249 15358 tablet_copy_service.cc:342] P e494ca2918594ba792c2d37d98b8c607: Request end of tablet copy session 0d9d84f0f3224d1a8fbaedd1021d730f-efbf0e49f78a4ff099786f47eedc3c8b received from {username='slave'} at 127.14.106.200:39839
I20240501 20:08:52.766386 15358 tablet_copy_service.cc:433] P e494ca2918594ba792c2d37d98b8c607: ending tablet copy session 0d9d84f0f3224d1a8fbaedd1021d730f-efbf0e49f78a4ff099786f47eedc3c8b on tablet efbf0e49f78a4ff099786f47eedc3c8b with peer 0d9d84f0f3224d1a8fbaedd1021d730f
W20240501 20:08:52.806298 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:08:52.835155 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:08:52.922849 16442 raft_consensus.cc:1203] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 20:08:53.196625 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:08:53.240239 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:08:53.253501 16834 raft_consensus.cc:1203] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20240501 20:08:53.283205 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:08:53.301532 17929 raft_consensus.cc:1050] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d: attempting to promote NON_VOTER 250badc514c94bcdbe2ace1b62a0874f to VOTER
I20240501 20:08:53.302513 17929 consensus_queue.cc:237] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:08:53.305141 16703 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:08:53.305122 16442 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f [term 1 LEARNER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:08:53.305187 15078 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:08:53.305867 17881 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:53.306262 17931 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:53.306603 17929 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:53.309657 17929 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:53.310338 16703 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
W20240501 20:08:53.311944 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:08:53.314677 16442 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:53.314673 15078 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:53.315778 14803 catalog_manager.cc:5484] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d reported cstate change: config changed from index 7 to 8, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:08:53.322875 15206 consensus_queue.cc:237] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:08:53.325066 16703 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:08:53.325078 16442 raft_consensus.cc:1261] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f [term 1 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:08:53.325691 17931 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:08:53.325982 17929 consensus_queue.cc:1035] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:08:53.328289 17931 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 2177fb70b1324acb81b2bea7be19955d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:53.328756 16703 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:53.330983 16442 raft_consensus.cc:2932] T 07797a04ba4b4ee3801855bfdbc1720d P 250badc514c94bcdbe2ace1b62a0874f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:53.334347 14792 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 07797a04ba4b4ee3801855bfdbc1720d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:53.334396 14805 catalog_manager.cc:5484] T 07797a04ba4b4ee3801855bfdbc1720d P 8dddd9d9b8fb47cb944ad4179152a915 reported cstate change: config changed from index 8 to 9, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "8dddd9d9b8fb47cb944ad4179152a915" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 38431 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:08:53.339497 15057 tablet_service.cc:1514] Processing DeleteTablet for tablet 07797a04ba4b4ee3801855bfdbc1720d with delete_type TABLET_DATA_TOMBSTONED (TS 4ed2ac58f951430a8c7e084f97d7a2a4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60194
I20240501 20:08:53.340267 17942 tablet_replica.cc:330] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: stopping tablet replica
I20240501 20:08:53.340449 17942 raft_consensus.cc:2229] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:53.340785 17942 raft_consensus.cc:2258] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:53.341565 17942 ts_tablet_manager.cc:1875] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:53.346904 17942 ts_tablet_manager.cc:1888] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:08:53.347018 17942 log.cc:1196] T 07797a04ba4b4ee3801855bfdbc1720d P 4ed2ac58f951430a8c7e084f97d7a2a4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/07797a04ba4b4ee3801855bfdbc1720d
I20240501 20:08:53.347685 14793 catalog_manager.cc:4830] TS 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059): tablet 07797a04ba4b4ee3801855bfdbc1720d (table rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]) successfully deleted
W20240501 20:08:53.378741 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:08:53.392899 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:08:53.868495 17967 raft_consensus.cc:1050] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: attempting to promote NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f to VOTER
I20240501 20:08:53.869220 17967 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:08:53.871580 16834 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 LEARNER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:08:53.871845 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:08:53.872390 17967 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:08:53.872521 15078 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:08:53.872790 17966 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:08:53.873157 17969 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:08:53.884008 17966 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:53.884716 16442 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:53.885032 15078 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:53.888703 16834 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:53.889551 14805 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 reported cstate change: config changed from index 8 to 9, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:08:54.008459 15338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b"
dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49310
I20240501 20:08:54.008704 15338 raft_consensus.cc:595] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Received request to transfer leadership
I20240501 20:08:54.226785 15338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b"
dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49332
I20240501 20:08:54.226980 15338 raft_consensus.cc:595] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Received request to transfer leadership
I20240501 20:08:54.306326 17969 raft_consensus.cc:979] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: : Instructing follower 0d9d84f0f3224d1a8fbaedd1021d730f to start an election
I20240501 20:08:54.306521 17969 raft_consensus.cc:1067] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Signalling peer 0d9d84f0f3224d1a8fbaedd1021d730f to start an election
I20240501 20:08:54.307317 16834 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b"
dest_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f"
from {username='slave'} at 127.14.106.196:39675
I20240501 20:08:54.307577 16834 raft_consensus.cc:482] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:54.307749 16834 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:08:54.309692 16834 raft_consensus.cc:504] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:08:54.311250 16834 leader_election.cc:288] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [CANDIDATE]: Term 3 election: Requested vote from peers 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059), e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197:35831)
I20240501 20:08:54.312217 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:54.312520 15338 raft_consensus.cc:3032] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:08:54.312614 15338 raft_consensus.cc:729] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Running, Role: LEADER
I20240501 20:08:54.313045 15338 consensus_queue.cc:260] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:08:54.313642 15338 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:08:54.321672 15338 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9d84f0f3224d1a8fbaedd1021d730f in term 3.
I20240501 20:08:54.329682 16442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "250badc514c94bcdbe2ace1b62a0874f"
I20240501 20:08:54.329682 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbf0e49f78a4ff099786f47eedc3c8b" candidate_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:08:54.329895 15078 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:08:54.329896 16442 raft_consensus.cc:3037] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:08:54.331826 16442 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9d84f0f3224d1a8fbaedd1021d730f in term 3.
I20240501 20:08:54.331965 15078 raft_consensus.cc:2454] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9d84f0f3224d1a8fbaedd1021d730f in term 3.
I20240501 20:08:54.332360 16769 leader_election.cc:302] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d9d84f0f3224d1a8fbaedd1021d730f, 250badc514c94bcdbe2ace1b62a0874f, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:54.332758 17976 raft_consensus.cc:2781] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:08:54.333315 17976 raft_consensus.cc:686] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 LEADER]: Becoming Leader. State: Replica: 0d9d84f0f3224d1a8fbaedd1021d730f, State: Running, Role: LEADER
I20240501 20:08:54.333824 17976 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:08:54.337888 14802 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: term changed from 2 to 3, leader changed from e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) to 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200). New cstate: current_term: 3 leader_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:08:54.722033 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:08:54.759884 15338 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 20:08:54.760542 17976 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:08:54.827960 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 20:08:54.828657 17976 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:08:54.837078 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:08:54.846357 15078 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 20:08:54.848410 18018 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:08:54.912971 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:08:55.379525 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:08:55.446301 16834 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:08:55.448057 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 20:08:55.448081 15078 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 20:08:55.448633 18038 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:08:55.448935 18037 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:08:55.451079 18036 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:55.451575 15078 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:55.451750 16442 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:08:55.454721 14790 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efbf0e49f78a4ff099786f47eedc3c8b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:55.455952 14803 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 9 to 11, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New cstate: current_term: 3 leader_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:08:55.461643 15317 tablet_service.cc:1514] Processing DeleteTablet for tablet efbf0e49f78a4ff099786f47eedc3c8b with delete_type TABLET_DATA_TOMBSTONED (TS e494ca2918594ba792c2d37d98b8c607 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44314
I20240501 20:08:55.462442 18045 tablet_replica.cc:330] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: stopping tablet replica
I20240501 20:08:55.462667 18045 raft_consensus.cc:2229] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:55.462917 18045 raft_consensus.cc:2258] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:55.463780 18045 ts_tablet_manager.cc:1875] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:55.468081 18045 ts_tablet_manager.cc:1888] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240501 20:08:55.468222 18045 log.cc:1196] T efbf0e49f78a4ff099786f47eedc3c8b P e494ca2918594ba792c2d37d98b8c607: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/efbf0e49f78a4ff099786f47eedc3c8b
I20240501 20:08:55.469043 14791 catalog_manager.cc:4830] TS e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333): tablet efbf0e49f78a4ff099786f47eedc3c8b (table rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]) successfully deleted
W20240501 20:08:55.503225 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:08:55.508085 15338 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:55.510701 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:08:55.510887 15206 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:08:55.511301 17967 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:08:55.511646 18059 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:08:55.514811 17967 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:08:55.516467 15206 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:08:55.518383 16834 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:08:55.522791 14802 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 6 to 7, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
W20240501 20:08:55.523505 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:08:55.540459 15271 consensus_peers.cc:485] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 -> Peer 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031): Couldn't send request to peer 98d6b3fd85674334a24cd273bf525324. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e093bed4aae0484faa3cba7a4cc52ff9. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:08:55.590569 15271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:08:55.591490 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:08:55.745390 15011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:08:55.746348 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:08:55.770689 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:08:55.814380 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:08:55.989609 18078 ts_tablet_manager.cc:902] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: Initiating tablet copy from peer e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333)
I20240501 20:08:55.991468 18078 tablet_copy_client.cc:308] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:39333
I20240501 20:08:55.998275 15358 tablet_copy_service.cc:140] P e494ca2918594ba792c2d37d98b8c607: Received BeginTabletCopySession request for tablet e093bed4aae0484faa3cba7a4cc52ff9 from peer 98d6b3fd85674334a24cd273bf525324 ({username='slave'} at 127.14.106.198:42487)
I20240501 20:08:55.998435 15358 tablet_copy_service.cc:161] P e494ca2918594ba792c2d37d98b8c607: Beginning new tablet copy session on tablet e093bed4aae0484faa3cba7a4cc52ff9 from peer 98d6b3fd85674334a24cd273bf525324 at {username='slave'} at 127.14.106.198:42487: session id = 98d6b3fd85674334a24cd273bf525324-e093bed4aae0484faa3cba7a4cc52ff9
I20240501 20:08:56.000514 15358 tablet_copy_source_session.cc:205] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:56.001931 18078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e093bed4aae0484faa3cba7a4cc52ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:56.006220 18078 tablet_copy_client.cc:783] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:56.006477 18078 tablet_copy_client.cc:647] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:56.008034 18082 tablet_copy_client.cc:1002] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: tablet copy: Injecting latency into file download: 1500
W20240501 20:08:56.060590 15141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:08:56.153205 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:08:56.165038 14790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 20:08:56.166242 14790 catalog_manager.cc:4573] TS 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c28b52cf6d9d4fa289de88ef97e08bee: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:08:57.173305 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:08:57.363185 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:08:57.508411 18078 tablet_copy_client.cc:523] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:57.511041 18078 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: Bootstrap starting.
I20240501 20:08:57.545210 18078 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:57.545614 18078 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: Bootstrap complete.
I20240501 20:08:57.545881 18078 ts_tablet_manager.cc:1367] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: Time spent bootstrapping tablet: real 0.035s user 0.032s sys 0.000s
I20240501 20:08:57.546697 18078 raft_consensus.cc:348] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:57.546911 18078 raft_consensus.cc:729] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98d6b3fd85674334a24cd273bf525324, State: Initialized, Role: LEARNER
I20240501 20:08:57.547101 18078 consensus_queue.cc:260] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:08:57.547860 18078 ts_tablet_manager.cc:1398] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:08:57.548575 15358 tablet_copy_service.cc:342] P e494ca2918594ba792c2d37d98b8c607: Request end of tablet copy session 98d6b3fd85674334a24cd273bf525324-e093bed4aae0484faa3cba7a4cc52ff9 received from {username='slave'} at 127.14.106.198:42487
I20240501 20:08:57.548702 15358 tablet_copy_service.cc:433] P e494ca2918594ba792c2d37d98b8c607: ending tablet copy session 98d6b3fd85674334a24cd273bf525324-e093bed4aae0484faa3cba7a4cc52ff9 on tablet e093bed4aae0484faa3cba7a4cc52ff9 with peer 98d6b3fd85674334a24cd273bf525324
W20240501 20:08:57.602905 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:08:57.964428 16573 raft_consensus.cc:1203] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 20:08:58.014107 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:08:58.031004 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:08:58.083456 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:08:58.166807 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:08:58.363220 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:08:58.364449 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:08:58.464213 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:08:58.466084 18149 raft_consensus.cc:1050] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: attempting to promote NON_VOTER 98d6b3fd85674334a24cd273bf525324 to VOTER
I20240501 20:08:58.467052 18149 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:08:58.469741 16573 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 1 LEARNER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:08:58.470018 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:08:58.470314 18148 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:58.470433 15206 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Refusing update from remote peer e494ca2918594ba792c2d37d98b8c607: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:08:58.470860 18149 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:58.471221 18126 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:08:58.475822 18149 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:58.476699 15206 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:58.481813 16573 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:58.483461 14803 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 reported cstate change: config changed from index 7 to 8, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e494ca2918594ba792c2d37d98b8c607" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:08:58.494452 16834 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:58.510654 15338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9"
dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49468
I20240501 20:08:58.510891 15338 raft_consensus.cc:595] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Received request to transfer leadership
W20240501 20:08:58.606701 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:08:58.728387 15338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9"
dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49478
I20240501 20:08:58.728598 15338 raft_consensus.cc:595] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:58.946038 18149 raft_consensus.cc:979] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: : Instructing follower 2177fb70b1324acb81b2bea7be19955d to start an election
I20240501 20:08:58.946192 18149 raft_consensus.cc:1067] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Signalling peer 2177fb70b1324acb81b2bea7be19955d to start an election
I20240501 20:08:58.946866 15206 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9"
dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
from {username='slave'} at 127.14.106.196:55825
I20240501 20:08:58.947099 15206 raft_consensus.cc:482] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:58.947216 15206 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:58.948719 15338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9"
dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49494
I20240501 20:08:58.948925 15338 raft_consensus.cc:595] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:58.949604 15206 raft_consensus.cc:504] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:08:58.951053 15206 leader_election.cc:288] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 election: Requested vote from peers e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333), 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973), 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031)
I20240501 20:08:58.952384 15338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e494ca2918594ba792c2d37d98b8c607"
I20240501 20:08:58.952598 15338 raft_consensus.cc:3032] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:58.952672 15338 raft_consensus.cc:729] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e494ca2918594ba792c2d37d98b8c607, State: Running, Role: LEADER
I20240501 20:08:58.952986 15338 consensus_queue.cc:260] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:08:58.953474 15338 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:58.956808 15338 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 2.
I20240501 20:08:58.957674 16834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f"
I20240501 20:08:58.957962 16834 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:58.959322 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "2177fb70b1324acb81b2bea7be19955d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "98d6b3fd85674334a24cd273bf525324"
I20240501 20:08:58.959578 16573 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:58.960216 16834 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 2.
I20240501 20:08:58.960935 15141 leader_election.cc:302] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d9d84f0f3224d1a8fbaedd1021d730f, 2177fb70b1324acb81b2bea7be19955d, e494ca2918594ba792c2d37d98b8c607; no voters:
I20240501 20:08:58.961226 17931 raft_consensus.cc:2781] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:58.961684 17931 raft_consensus.cc:686] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Becoming Leader. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Running, Role: LEADER
I20240501 20:08:58.962006 17931 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:08:58.962291 16573 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2177fb70b1324acb81b2bea7be19955d in term 2.
I20240501 20:08:58.964483 14802 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d reported cstate change: term changed from 1 to 2, leader changed from e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) to 2177fb70b1324acb81b2bea7be19955d (127.14.106.195). New cstate: current_term: 2 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:59.409116 15338 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:08:59.409665 17931 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:08:59.440021 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:08:59.440577 17931 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:08:59.444881 16573 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:08:59.445672 17931 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:08:59.753235 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:08:59.756986 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:08:59.852937 15206 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:08:59.854841 16573 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:08:59.854874 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:08:59.855396 18207 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:08:59.855741 17931 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:08:59.858217 18193 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:59.858826 16834 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:59.859189 16573 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:59.862478 14792 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e093bed4aae0484faa3cba7a4cc52ff9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:59.863761 14802 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 8 to 10, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New cstate: current_term: 2 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:08:59.868132 15317 tablet_service.cc:1514] Processing DeleteTablet for tablet e093bed4aae0484faa3cba7a4cc52ff9 with delete_type TABLET_DATA_TOMBSTONED (TS e494ca2918594ba792c2d37d98b8c607 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44314
I20240501 20:08:59.868834 18225 tablet_replica.cc:330] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: stopping tablet replica
I20240501 20:08:59.869050 18225 raft_consensus.cc:2229] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:59.869264 18225 raft_consensus.cc:2258] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:59.870057 18225 ts_tablet_manager.cc:1875] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:59.875111 18225 ts_tablet_manager.cc:1888] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:08:59.875209 18225 log.cc:1196] T e093bed4aae0484faa3cba7a4cc52ff9 P e494ca2918594ba792c2d37d98b8c607: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/e093bed4aae0484faa3cba7a4cc52ff9
I20240501 20:08:59.875916 14791 catalog_manager.cc:4830] TS e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333): tablet e093bed4aae0484faa3cba7a4cc52ff9 (table rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]) successfully deleted
W20240501 20:08:59.895700 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:09:00.014950 15078 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:09:00.017629 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:09:00.018273 15338 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:09:00.018282 18238 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:09:00.018949 18211 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:09:00.021243 18238 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
I20240501 20:09:00.023298 16573 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
I20240501 20:09:00.024608 15338 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } } }
W20240501 20:09:00.027717 15011 consensus_peers.cc:485] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973): Couldn't send request to peer 0d9d84f0f3224d1a8fbaedd1021d730f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c28b52cf6d9d4fa289de88ef97e08bee. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:00.028417 14803 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: config changed from index 8 to 9, NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) added. New cstate: current_term: 2 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:00.474555 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:09:00.607539 18257 ts_tablet_manager.cc:902] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: Initiating tablet copy from peer 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
W20240501 20:09:00.608197 15271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 20:09:00.609104 18257 tablet_copy_client.cc:308] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:34059
W20240501 20:09:00.609316 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:09:00.610030 15098 tablet_copy_service.cc:140] P 4ed2ac58f951430a8c7e084f97d7a2a4: Received BeginTabletCopySession request for tablet c28b52cf6d9d4fa289de88ef97e08bee from peer 0d9d84f0f3224d1a8fbaedd1021d730f ({username='slave'} at 127.14.106.200:33065)
I20240501 20:09:00.610210 15098 tablet_copy_service.cc:161] P 4ed2ac58f951430a8c7e084f97d7a2a4: Beginning new tablet copy session on tablet c28b52cf6d9d4fa289de88ef97e08bee from peer 0d9d84f0f3224d1a8fbaedd1021d730f at {username='slave'} at 127.14.106.200:33065: session id = 0d9d84f0f3224d1a8fbaedd1021d730f-c28b52cf6d9d4fa289de88ef97e08bee
I20240501 20:09:00.612941 15098 tablet_copy_source_session.cc:205] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:00.614430 18257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b52cf6d9d4fa289de88ef97e08bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.619153 18257 tablet_copy_client.cc:783] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:00.619393 18257 tablet_copy_client.cc:647] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:00.621090 18261 tablet_copy_client.cc:1002] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Injecting latency into file download: 1500
W20240501 20:09:00.731128 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:09:00.740128 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:09:00.864770 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:09:00.994063 15011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:09:00.994905 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:09:01.003916 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:09:01.097324 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:09:01.304809 15141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:09:02.121510 18257 tablet_copy_client.cc:523] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:02.124258 18257 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap starting.
I20240501 20:09:02.161108 18257 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:02.161521 18257 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: Bootstrap complete.
I20240501 20:09:02.161811 18257 ts_tablet_manager.cc:1367] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: Time spent bootstrapping tablet: real 0.038s user 0.030s sys 0.005s
I20240501 20:09:02.162519 18257 raft_consensus.cc:348] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:09:02.162758 18257 raft_consensus.cc:729] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0d9d84f0f3224d1a8fbaedd1021d730f, State: Initialized, Role: LEARNER
I20240501 20:09:02.162979 18257 consensus_queue.cc:260] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: NON_VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: true } }
I20240501 20:09:02.163499 18257 ts_tablet_manager.cc:1398] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:09:02.164347 15098 tablet_copy_service.cc:342] P 4ed2ac58f951430a8c7e084f97d7a2a4: Request end of tablet copy session 0d9d84f0f3224d1a8fbaedd1021d730f-c28b52cf6d9d4fa289de88ef97e08bee received from {username='slave'} at 127.14.106.200:33065
I20240501 20:09:02.164506 15098 tablet_copy_service.cc:433] P 4ed2ac58f951430a8c7e084f97d7a2a4: ending tablet copy session 0d9d84f0f3224d1a8fbaedd1021d730f-c28b52cf6d9d4fa289de88ef97e08bee on tablet c28b52cf6d9d4fa289de88ef97e08bee with peer 0d9d84f0f3224d1a8fbaedd1021d730f
W20240501 20:09:02.292920 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:09:02.319177 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:09:02.361089 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:09:02.499114 16834 raft_consensus.cc:1203] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240501 20:09:02.971120 18323 raft_consensus.cc:1050] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: attempting to promote NON_VOTER 0d9d84f0f3224d1a8fbaedd1021d730f to VOTER
I20240501 20:09:02.972180 18323 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:09:02.975494 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:09:02.975876 15338 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:09:02.976078 16834 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 LEARNER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:09:02.976579 18324 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:09:02.976974 18290 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:09:02.977630 18290 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:09:02.997352 18331 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:09:02.998350 16834 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:09:03.002081 16573 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:09:03.003320 14802 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 9 to 10, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
W20240501 20:09:03.014853 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:09:03.015556 15338 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:09:03.018114 15078 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } }
I20240501 20:09:03.019750 16834 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:09:03.019909 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:09:03.020462 18324 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:09:03.020916 18331 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:09:03.023042 18324 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:09:03.023752 16573 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:09:03.024681 16834 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } }
I20240501 20:09:03.027485 14793 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c28b52cf6d9d4fa289de88ef97e08bee with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:03.029035 14805 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: config changed from index 10 to 11, VOTER e494ca2918594ba792c2d37d98b8c607 (127.14.106.196) evicted. New cstate: current_term: 2 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:03.032382 15317 tablet_service.cc:1514] Processing DeleteTablet for tablet c28b52cf6d9d4fa289de88ef97e08bee with delete_type TABLET_DATA_TOMBSTONED (TS e494ca2918594ba792c2d37d98b8c607 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44314
I20240501 20:09:03.033638 18341 tablet_replica.cc:330] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: stopping tablet replica
I20240501 20:09:03.033856 18341 raft_consensus.cc:2229] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:03.034181 18341 raft_consensus.cc:2258] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:03.035039 18341 ts_tablet_manager.cc:1875] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:03.039512 18341 ts_tablet_manager.cc:1888] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 20:09:03.039611 18341 log.cc:1196] T c28b52cf6d9d4fa289de88ef97e08bee P e494ca2918594ba792c2d37d98b8c607: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/c28b52cf6d9d4fa289de88ef97e08bee
I20240501 20:09:03.040246 14791 catalog_manager.cc:4830] TS e494ca2918594ba792c2d37d98b8c607 (127.14.106.196:39333): tablet c28b52cf6d9d4fa289de88ef97e08bee (table rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]) successfully deleted
W20240501 20:09:03.112382 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:09:03.124439 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:09:03.208873 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:09:03.311079 16814 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:09:03.315196 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:09:03.327180 15057 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:03.333235 15188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:03.337574 16422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:03.354979 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:03.384966 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:03.407284 15317 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:09:03.439934 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:09:03.475342 16834 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:09:03.477941 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 20:09:03.478560 15078 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 20:09:03.478742 18390 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:09:03.479234 18123 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:09:03.481999 18390 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:09:03.483212 15078 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
I20240501 20:09:03.486196 16442 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } } }
W20240501 20:09:03.487273 16767 consensus_peers.cc:485] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f -> Peer 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031): Couldn't send request to peer 98d6b3fd85674334a24cd273bf525324. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efbf0e49f78a4ff099786f47eedc3c8b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:03.492684 14803 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 11 to 12, NON_VOTER 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) added. New cstate: current_term: 3 leader_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:03.496868 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:09:03.516096 15206 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:09:03.519253 16573 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20240501 20:09:03.519388 15143 consensus_peers.cc:485] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d -> Peer 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197:35831): Couldn't send request to peer 250badc514c94bcdbe2ace1b62a0874f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e093bed4aae0484faa3cba7a4cc52ff9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:03.519603 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:09:03.520177 18402 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:09:03.520557 17931 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:09:03.523598 18402 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:09:03.524219 16834 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:09:03.530297 14803 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 10 to 11, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New cstate: current_term: 2 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:09:03.545361 16573 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
W20240501 20:09:03.789734 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:09:03.878350 18428 ts_tablet_manager.cc:902] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: Initiating tablet copy from peer 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973)
I20240501 20:09:03.879827 18428 tablet_copy_client.cc:308] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.200:40973
I20240501 20:09:03.885654 16854 tablet_copy_service.cc:140] P 0d9d84f0f3224d1a8fbaedd1021d730f: Received BeginTabletCopySession request for tablet efbf0e49f78a4ff099786f47eedc3c8b from peer 98d6b3fd85674334a24cd273bf525324 ({username='slave'} at 127.14.106.198:38837)
I20240501 20:09:03.885892 16854 tablet_copy_service.cc:161] P 0d9d84f0f3224d1a8fbaedd1021d730f: Beginning new tablet copy session on tablet efbf0e49f78a4ff099786f47eedc3c8b from peer 98d6b3fd85674334a24cd273bf525324 at {username='slave'} at 127.14.106.198:38837: session id = 98d6b3fd85674334a24cd273bf525324-efbf0e49f78a4ff099786f47eedc3c8b
I20240501 20:09:03.888928 16854 tablet_copy_source_session.cc:205] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:03.890455 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbf0e49f78a4ff099786f47eedc3c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:03.894773 18428 tablet_copy_client.cc:783] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:03.895017 18428 tablet_copy_client.cc:647] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:03.896754 18432 tablet_copy_client.cc:1002] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: tablet copy: Injecting latency into file download: 1500
I20240501 20:09:03.991883 18439 ts_tablet_manager.cc:902] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: Initiating tablet copy from peer 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095)
I20240501 20:09:03.999636 18439 tablet_copy_client.cc:308] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:34095
I20240501 20:09:04.000278 15227 tablet_copy_service.cc:140] P 2177fb70b1324acb81b2bea7be19955d: Received BeginTabletCopySession request for tablet e093bed4aae0484faa3cba7a4cc52ff9 from peer 250badc514c94bcdbe2ace1b62a0874f ({username='slave'} at 127.14.106.197:38403)
I20240501 20:09:04.000432 15227 tablet_copy_service.cc:161] P 2177fb70b1324acb81b2bea7be19955d: Beginning new tablet copy session on tablet e093bed4aae0484faa3cba7a4cc52ff9 from peer 250badc514c94bcdbe2ace1b62a0874f at {username='slave'} at 127.14.106.197:38403: session id = 250badc514c94bcdbe2ace1b62a0874f-e093bed4aae0484faa3cba7a4cc52ff9
I20240501 20:09:04.003157 15227 tablet_copy_source_session.cc:205] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:04.004390 18439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e093bed4aae0484faa3cba7a4cc52ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.009050 18439 tablet_copy_client.cc:783] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:04.009346 18439 tablet_copy_client.cc:647] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:04.010782 18448 tablet_copy_client.cc:1002] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Injecting latency into file download: 1500
W20240501 20:09:04.371701 14790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:09:04.372761 14790 catalog_manager.cc:4573] TS 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c28b52cf6d9d4fa289de88ef97e08bee: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111)
W20240501 20:09:04.780710 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:09:04.845031 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:09:04.888224 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:09:05.397118 18428 tablet_copy_client.cc:523] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:05.399775 18428 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: Bootstrap starting.
I20240501 20:09:05.433310 18428 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:05.433748 18428 tablet_bootstrap.cc:492] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: Bootstrap complete.
I20240501 20:09:05.434022 18428 ts_tablet_manager.cc:1367] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: Time spent bootstrapping tablet: real 0.034s user 0.028s sys 0.003s
I20240501 20:09:05.434686 18428 raft_consensus.cc:348] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:09:05.434909 18428 raft_consensus.cc:729] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 98d6b3fd85674334a24cd273bf525324, State: Initialized, Role: LEARNER
I20240501 20:09:05.435118 18428 consensus_queue.cc:260] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:09:05.435753 18428 ts_tablet_manager.cc:1398] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:09:05.436583 16854 tablet_copy_service.cc:342] P 0d9d84f0f3224d1a8fbaedd1021d730f: Request end of tablet copy session 98d6b3fd85674334a24cd273bf525324-efbf0e49f78a4ff099786f47eedc3c8b received from {username='slave'} at 127.14.106.198:38837
I20240501 20:09:05.436765 16854 tablet_copy_service.cc:433] P 0d9d84f0f3224d1a8fbaedd1021d730f: ending tablet copy session 98d6b3fd85674334a24cd273bf525324-efbf0e49f78a4ff099786f47eedc3c8b on tablet efbf0e49f78a4ff099786f47eedc3c8b with peer 98d6b3fd85674334a24cd273bf525324
I20240501 20:09:05.451659 16573 raft_consensus.cc:1203] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.11->[3.12-3.12] Dedup: 3.12->[]
I20240501 20:09:05.511240 18439 tablet_copy_client.cc:523] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:05.514104 18439 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap starting.
I20240501 20:09:05.561133 18439 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:05.561553 18439 tablet_bootstrap.cc:492] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap complete.
I20240501 20:09:05.561836 18439 ts_tablet_manager.cc:1367] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: Time spent bootstrapping tablet: real 0.048s user 0.039s sys 0.008s
I20240501 20:09:05.562680 18439 raft_consensus.cc:348] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:09:05.562911 18439 raft_consensus.cc:729] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 250badc514c94bcdbe2ace1b62a0874f, State: Initialized, Role: LEARNER
I20240501 20:09:05.563060 18439 consensus_queue.cc:260] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:09:05.563741 18439 ts_tablet_manager.cc:1398] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:05.564540 15227 tablet_copy_service.cc:342] P 2177fb70b1324acb81b2bea7be19955d: Request end of tablet copy session 250badc514c94bcdbe2ace1b62a0874f-e093bed4aae0484faa3cba7a4cc52ff9 received from {username='slave'} at 127.14.106.197:38403
I20240501 20:09:05.564661 15227 tablet_copy_service.cc:433] P 2177fb70b1324acb81b2bea7be19955d: ending tablet copy session 250badc514c94bcdbe2ace1b62a0874f-e093bed4aae0484faa3cba7a4cc52ff9 on tablet e093bed4aae0484faa3cba7a4cc52ff9 with peer 250badc514c94bcdbe2ace1b62a0874f
W20240501 20:09:05.578765 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:09:05.662953 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:09:05.671885 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:09:05.772519 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:09:05.849520 15271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:09:05.873509 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:09:05.884379 18545 raft_consensus.cc:1050] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f: attempting to promote NON_VOTER 98d6b3fd85674334a24cd273bf525324 to VOTER
I20240501 20:09:05.885490 18545 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:09:05.888099 16573 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324 [term 3 LEARNER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 20:09:05.888289 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 20:09:05.888303 15078 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 20:09:05.888710 18545 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:09:05.889089 18487 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:09:05.889379 18548 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:09:05.892680 18545 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:09:05.893307 15078 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:09:05.893404 16442 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:09:05.896073 16573 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:09:05.897136 14802 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 12 to 13, 98d6b3fd85674334a24cd273bf525324 (127.14.106.198) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:05.904685 16834 consensus_queue.cc:237] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } }
I20240501 20:09:05.906559 16442 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 20:09:05.906781 16573 raft_consensus.cc:1261] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Refusing update from remote peer 0d9d84f0f3224d1a8fbaedd1021d730f: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 20:09:05.907486 18545 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:09:05.907825 18548 consensus_queue.cc:1035] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:09:05.909833 18545 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:09:05.910504 16442 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:09:05.911127 16573 raft_consensus.cc:2932] T efbf0e49f78a4ff099786f47eedc3c8b P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } }
I20240501 20:09:05.912928 14790 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efbf0e49f78a4ff099786f47eedc3c8b with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:05.914315 14803 catalog_manager.cc:5484] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 13 to 14, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New cstate: current_term: 3 leader_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:05.918174 15057 tablet_service.cc:1514] Processing DeleteTablet for tablet efbf0e49f78a4ff099786f47eedc3c8b with delete_type TABLET_DATA_TOMBSTONED (TS 4ed2ac58f951430a8c7e084f97d7a2a4 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:60194
I20240501 20:09:05.919116 18556 tablet_replica.cc:330] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: stopping tablet replica
I20240501 20:09:05.919354 18556 raft_consensus.cc:2229] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:05.919706 18556 raft_consensus.cc:2258] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:05.920512 18556 ts_tablet_manager.cc:1875] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:05.925024 18556 ts_tablet_manager.cc:1888] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240501 20:09:05.925148 18556 log.cc:1196] T efbf0e49f78a4ff099786f47eedc3c8b P 4ed2ac58f951430a8c7e084f97d7a2a4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/efbf0e49f78a4ff099786f47eedc3c8b
I20240501 20:09:05.925907 14793 catalog_manager.cc:4830] TS 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059): tablet efbf0e49f78a4ff099786f47eedc3c8b (table rebalance_test_table_3 [id=7c835e2dfff94c388d06a80055d0c943]) successfully deleted
W20240501 20:09:05.947175 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:09:06.023495 16442 raft_consensus.cc:1203] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240501 20:09:06.068989 15011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:09:06.176605 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:09:06.476393 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:09:06.559365 15141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:09:06.631379 18499 raft_consensus.cc:1050] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: attempting to promote NON_VOTER 250badc514c94bcdbe2ace1b62a0874f to VOTER
I20240501 20:09:06.632341 18499 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:06.635187 16442 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 2 LEARNER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:09:06.635545 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:09:06.636170 16573 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 2177fb70b1324acb81b2bea7be19955d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:09:06.636170 18583 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:09:06.636718 18572 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:09:06.637632 18583 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:09:06.646400 18590 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:06.647190 16573 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:06.647522 16442 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:06.650828 16834 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:06.651882 14805 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d reported cstate change: config changed from index 11 to 12, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2177fb70b1324acb81b2bea7be19955d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:06.862884 15206 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9"
dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54176
I20240501 20:09:06.863106 15206 raft_consensus.cc:595] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Received request to transfer leadership
I20240501 20:09:07.046834 18572 raft_consensus.cc:979] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: : Instructing follower 250badc514c94bcdbe2ace1b62a0874f to start an election
I20240501 20:09:07.047058 18572 raft_consensus.cc:1067] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Signalling peer 250badc514c94bcdbe2ace1b62a0874f to start an election
I20240501 20:09:07.047741 16442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9"
dest_uuid: "250badc514c94bcdbe2ace1b62a0874f"
from {username='slave'} at 127.14.106.195:46659
I20240501 20:09:07.048034 16442 raft_consensus.cc:482] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:07.048144 16442 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:07.050055 16442 raft_consensus.cc:504] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:07.051604 16442 leader_election.cc:288] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [CANDIDATE]: Term 3 election: Requested vote from peers 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095), 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973), 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031)
I20240501 20:09:07.052084 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "250badc514c94bcdbe2ace1b62a0874f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "2177fb70b1324acb81b2bea7be19955d"
I20240501 20:09:07.052346 15206 raft_consensus.cc:3032] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:07.052474 15206 raft_consensus.cc:729] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2177fb70b1324acb81b2bea7be19955d, State: Running, Role: LEADER
I20240501 20:09:07.052840 15206 consensus_queue.cc:260] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:07.053448 15206 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:07.057431 15206 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250badc514c94bcdbe2ace1b62a0874f in term 3.
I20240501 20:09:07.064183 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "250badc514c94bcdbe2ace1b62a0874f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "98d6b3fd85674334a24cd273bf525324"
I20240501 20:09:07.064419 16573 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:07.064952 16834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e093bed4aae0484faa3cba7a4cc52ff9" candidate_uuid: "250badc514c94bcdbe2ace1b62a0874f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f"
I20240501 20:09:07.065158 16834 raft_consensus.cc:3037] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:07.066692 16573 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250badc514c94bcdbe2ace1b62a0874f in term 3.
I20240501 20:09:07.067234 16375 leader_election.cc:302] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2177fb70b1324acb81b2bea7be19955d, 250badc514c94bcdbe2ace1b62a0874f, 98d6b3fd85674334a24cd273bf525324; no voters:
I20240501 20:09:07.067610 18595 raft_consensus.cc:2781] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:07.068058 18595 raft_consensus.cc:686] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 3 LEADER]: Becoming Leader. State: Replica: 250badc514c94bcdbe2ace1b62a0874f, State: Running, Role: LEADER
I20240501 20:09:07.068594 18595 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:07.068768 16834 raft_consensus.cc:2454] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250badc514c94bcdbe2ace1b62a0874f in term 3.
I20240501 20:09:07.072093 14802 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f reported cstate change: term changed from 2 to 3, leader changed from 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) to 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197). New cstate: current_term: 3 leader_uuid: "250badc514c94bcdbe2ace1b62a0874f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:07.428494 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:09:07.452991 15206 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 20:09:07.453650 18595 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 20:09:07.476212 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:09:07.611793 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 20:09:07.612411 18595 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:09:07.616804 16573 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 20:09:07.617404 18630 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 20:09:07.648277 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:09:07.986888 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:09:08.093067 16442 consensus_queue.cc:237] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:08.094843 16834 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 20:09:08.094923 16573 raft_consensus.cc:1261] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 20:09:08.095371 18632 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:09:08.095743 18630 consensus_queue.cc:1035] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:09:08.097862 18632 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:08.098606 16834 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:08.098937 16573 raft_consensus.cc:2932] T e093bed4aae0484faa3cba7a4cc52ff9 P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:08.103503 14802 catalog_manager.cc:5484] T e093bed4aae0484faa3cba7a4cc52ff9 P 250badc514c94bcdbe2ace1b62a0874f reported cstate change: config changed from index 12 to 14, VOTER 2177fb70b1324acb81b2bea7be19955d (127.14.106.195) evicted. New cstate: current_term: 3 leader_uuid: "250badc514c94bcdbe2ace1b62a0874f" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.104229 14792 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e093bed4aae0484faa3cba7a4cc52ff9 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:08.109407 15188 tablet_service.cc:1514] Processing DeleteTablet for tablet e093bed4aae0484faa3cba7a4cc52ff9 with delete_type TABLET_DATA_TOMBSTONED (TS 2177fb70b1324acb81b2bea7be19955d not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:34738
I20240501 20:09:08.111594 18652 tablet_replica.cc:330] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: stopping tablet replica
I20240501 20:09:08.111786 18652 raft_consensus.cc:2229] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.111976 18652 raft_consensus.cc:2258] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.112936 18652 ts_tablet_manager.cc:1875] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.119210 18652 ts_tablet_manager.cc:1888] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240501 20:09:08.119340 18652 log.cc:1196] T e093bed4aae0484faa3cba7a4cc52ff9 P 2177fb70b1324acb81b2bea7be19955d: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/e093bed4aae0484faa3cba7a4cc52ff9
I20240501 20:09:08.120071 14792 catalog_manager.cc:4830] TS 2177fb70b1324acb81b2bea7be19955d (127.14.106.195:34095): tablet e093bed4aae0484faa3cba7a4cc52ff9 (table rebalance_test_table_0 [id=19b9183513d2453480bca13639ad7d40]) successfully deleted
I20240501 20:09:08.145388 15078 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
W20240501 20:09:08.145685 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:09:08.147974 16834 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:09:08.148442 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:09:08.149367 18665 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240501 20:09:08.149705 18331 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:09:08.152998 18665 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
W20240501 20:09:08.153659 15013 consensus_peers.cc:485] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197:35831): Couldn't send request to peer 250badc514c94bcdbe2ace1b62a0874f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c28b52cf6d9d4fa289de88ef97e08bee. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:08.154898 16834 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:09:08.155359 16573 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } } }
I20240501 20:09:08.159197 14806 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 reported cstate change: config changed from index 11 to 12, NON_VOTER 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) added. New cstate: current_term: 2 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:08.200289 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:09:08.245765 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:09:08.449373 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:09:08.458570 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:09:08.641286 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:09:08.727363 18686 ts_tablet_manager.cc:902] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: Initiating tablet copy from peer 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059)
I20240501 20:09:08.728678 18686 tablet_copy_client.cc:308] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:34059
I20240501 20:09:08.738857 15098 tablet_copy_service.cc:140] P 4ed2ac58f951430a8c7e084f97d7a2a4: Received BeginTabletCopySession request for tablet c28b52cf6d9d4fa289de88ef97e08bee from peer 250badc514c94bcdbe2ace1b62a0874f ({username='slave'} at 127.14.106.197:60741)
I20240501 20:09:08.739032 15098 tablet_copy_service.cc:161] P 4ed2ac58f951430a8c7e084f97d7a2a4: Beginning new tablet copy session on tablet c28b52cf6d9d4fa289de88ef97e08bee from peer 250badc514c94bcdbe2ace1b62a0874f at {username='slave'} at 127.14.106.197:60741: session id = 250badc514c94bcdbe2ace1b62a0874f-c28b52cf6d9d4fa289de88ef97e08bee
I20240501 20:09:08.740945 15098 tablet_copy_source_session.cc:205] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:08.742203 18686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28b52cf6d9d4fa289de88ef97e08bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.746490 18686 tablet_copy_client.cc:783] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:08.746744 18686 tablet_copy_client.cc:647] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:08.748168 18690 tablet_copy_client.cc:1002] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Injecting latency into file download: 1500
W20240501 20:09:09.004063 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:09:09.733184 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:09:09.818467 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:09:09.987322 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:09:10.248548 18686 tablet_copy_client.cc:523] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:10.251092 18686 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap starting.
W20240501 20:09:10.273483 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:09:10.293530 18686 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:10.294051 18686 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: Bootstrap complete.
I20240501 20:09:10.294338 18686 ts_tablet_manager.cc:1367] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: Time spent bootstrapping tablet: real 0.043s user 0.031s sys 0.007s
I20240501 20:09:10.295094 18686 raft_consensus.cc:348] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:09:10.295305 18686 raft_consensus.cc:729] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 250badc514c94bcdbe2ace1b62a0874f, State: Initialized, Role: LEARNER
I20240501 20:09:10.295512 18686 consensus_queue.cc:260] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: true } }
I20240501 20:09:10.296461 18686 ts_tablet_manager.cc:1398] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:09:10.297863 15098 tablet_copy_service.cc:342] P 4ed2ac58f951430a8c7e084f97d7a2a4: Request end of tablet copy session 250badc514c94bcdbe2ace1b62a0874f-c28b52cf6d9d4fa289de88ef97e08bee received from {username='slave'} at 127.14.106.197:60741
I20240501 20:09:10.298002 15098 tablet_copy_service.cc:433] P 4ed2ac58f951430a8c7e084f97d7a2a4: ending tablet copy session 250badc514c94bcdbe2ace1b62a0874f-c28b52cf6d9d4fa289de88ef97e08bee on tablet c28b52cf6d9d4fa289de88ef97e08bee with peer 250badc514c94bcdbe2ace1b62a0874f
I20240501 20:09:10.400724 16442 raft_consensus.cc:1203] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240501 20:09:10.491596 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:09:10.601320 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:09:10.890748 15271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:09:10.891701 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:09:10.903918 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:09:10.968327 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:09:10.977557 18720 raft_consensus.cc:1050] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: attempting to promote NON_VOTER 250badc514c94bcdbe2ace1b62a0874f to VOTER
I20240501 20:09:10.978477 18720 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:10.981107 16442 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 2 LEARNER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:09:10.981154 16834 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:09:10.981796 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Refusing update from remote peer 4ed2ac58f951430a8c7e084f97d7a2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:09:10.981813 18754 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:09:10.982793 18754 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:09:10.983238 18720 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:09:10.986075 18720 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:10.986764 16442 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:10.987236 16573 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:10.991590 14805 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f reported cstate change: config changed from index 12 to 13, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:10.996757 16834 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
W20240501 20:09:11.113440 15011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:09:11.128037 15078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee"
dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54850
I20240501 20:09:11.128289 15078 raft_consensus.cc:595] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Received request to transfer leadership
W20240501 20:09:11.247047 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:09:11.346168 15078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee"
dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54870
I20240501 20:09:11.346419 15078 raft_consensus.cc:595] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Received request to transfer leadership
I20240501 20:09:11.372012 18720 raft_consensus.cc:979] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: : Instructing follower 250badc514c94bcdbe2ace1b62a0874f to start an election
I20240501 20:09:11.372246 18748 raft_consensus.cc:1067] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Signalling peer 250badc514c94bcdbe2ace1b62a0874f to start an election
I20240501 20:09:11.373116 16442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee"
dest_uuid: "250badc514c94bcdbe2ace1b62a0874f"
from {username='slave'} at 127.14.106.194:54009
I20240501 20:09:11.373327 16442 raft_consensus.cc:482] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:11.373467 16442 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:11.375416 16442 raft_consensus.cc:504] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:11.376132 16442 leader_election.cc:288] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [CANDIDATE]: Term 3 election: Requested vote from peers 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059), 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031), 0d9d84f0f3224d1a8fbaedd1021d730f (127.14.106.200:40973)
I20240501 20:09:11.376590 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "250badc514c94bcdbe2ace1b62a0874f" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4"
I20240501 20:09:11.376719 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "250badc514c94bcdbe2ace1b62a0874f" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "98d6b3fd85674334a24cd273bf525324"
I20240501 20:09:11.376812 15078 raft_consensus.cc:3032] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:11.376894 15078 raft_consensus.cc:729] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4ed2ac58f951430a8c7e084f97d7a2a4, State: Running, Role: LEADER
I20240501 20:09:11.376916 16573 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:11.376947 16834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28b52cf6d9d4fa289de88ef97e08bee" candidate_uuid: "250badc514c94bcdbe2ace1b62a0874f" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f"
I20240501 20:09:11.377163 16834 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:11.377224 15078 consensus_queue.cc:260] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:11.377727 15078 raft_consensus.cc:3037] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:11.378981 16573 raft_consensus.cc:2454] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250badc514c94bcdbe2ace1b62a0874f in term 3.
I20240501 20:09:11.379989 16834 raft_consensus.cc:2454] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250badc514c94bcdbe2ace1b62a0874f in term 3.
I20240501 20:09:11.380427 16375 leader_election.cc:302] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d9d84f0f3224d1a8fbaedd1021d730f, 250badc514c94bcdbe2ace1b62a0874f, 98d6b3fd85674334a24cd273bf525324; no voters:
I20240501 20:09:11.380728 18702 raft_consensus.cc:2781] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:11.381018 15078 raft_consensus.cc:2454] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250badc514c94bcdbe2ace1b62a0874f in term 3.
I20240501 20:09:11.381170 18702 raft_consensus.cc:686] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 3 LEADER]: Becoming Leader. State: Replica: 250badc514c94bcdbe2ace1b62a0874f, State: Running, Role: LEADER
I20240501 20:09:11.381527 18702 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:11.383931 14802 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f reported cstate change: term changed from 2 to 3, leader changed from 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) to 250badc514c94bcdbe2ace1b62a0874f (127.14.106.197). New cstate: current_term: 3 leader_uuid: "250badc514c94bcdbe2ace1b62a0874f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:11.505347 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:09:11.561990 15141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:09:11.843387 15078 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 20:09:11.843917 18702 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:09:11.894799 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 20:09:11.895388 18702 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:09:11.899664 16834 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 20:09:11.900925 18800 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 20:09:12.221204 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:09:12.329761 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:09:12.416072 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:09:12.515277 16442 consensus_queue.cc:237] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } }
I20240501 20:09:12.516824 16834 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 20:09:12.516999 16573 raft_consensus.cc:1261] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Refusing update from remote peer 250badc514c94bcdbe2ace1b62a0874f: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 20:09:12.517284 18821 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:09:12.517570 18822 consensus_queue.cc:1035] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:09:12.519804 18803 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 250badc514c94bcdbe2ace1b62a0874f [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:12.520440 16834 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:12.520422 16573 raft_consensus.cc:2932] T c28b52cf6d9d4fa289de88ef97e08bee P 98d6b3fd85674334a24cd273bf525324 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:12.524525 14792 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c28b52cf6d9d4fa289de88ef97e08bee with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:12.525732 14803 catalog_manager.cc:5484] T c28b52cf6d9d4fa289de88ef97e08bee P 0d9d84f0f3224d1a8fbaedd1021d730f reported cstate change: config changed from index 13 to 15, VOTER 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194) evicted. New cstate: current_term: 3 leader_uuid: "250badc514c94bcdbe2ace1b62a0874f" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0d9d84f0f3224d1a8fbaedd1021d730f" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "250badc514c94bcdbe2ace1b62a0874f" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35831 } attrs { promote: false } } }
I20240501 20:09:12.530606 15057 tablet_service.cc:1514] Processing DeleteTablet for tablet c28b52cf6d9d4fa289de88ef97e08bee with delete_type TABLET_DATA_TOMBSTONED (TS 4ed2ac58f951430a8c7e084f97d7a2a4 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:60194
I20240501 20:09:12.531510 18828 tablet_replica.cc:330] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: stopping tablet replica
I20240501 20:09:12.531703 18828 raft_consensus.cc:2229] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:12.531905 18828 raft_consensus.cc:2258] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:12.532703 18828 ts_tablet_manager.cc:1875] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:12.539588 18828 ts_tablet_manager.cc:1888] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240501 20:09:12.539710 18828 log.cc:1196] T c28b52cf6d9d4fa289de88ef97e08bee P 4ed2ac58f951430a8c7e084f97d7a2a4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/c28b52cf6d9d4fa289de88ef97e08bee
I20240501 20:09:12.540529 14793 catalog_manager.cc:4830] TS 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059): tablet c28b52cf6d9d4fa289de88ef97e08bee (table rebalance_test_table_2 [id=27826fa9ea7a4b13bb1c5ba44756ed9c]) successfully deleted
I20240501 20:09:12.707932 16814 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:12.719062 16422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.720137 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:12.746515 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:12.748991 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:12.760061 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:12.761979 15317 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:12.930796 16814 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:09:12.955575 15011 consensus_peers.cc:485] T 3931f8694a4a45ee81497f1c0b263d4d P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:09:12.967864 16422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.969683 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:12.994436 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:13.000221 15317 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:13.006264 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:13.020610 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:09:13.150121 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:09:13.187369 15011 consensus_peers.cc:485] T 6513ee10256249b8976bfeb4cf0bdbb7 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:09:13.256480 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.193:44871
--local_ip_for_outbound_sockets=127.14.106.193
--tserver_master_addrs=127.14.106.254:34075
--webserver_port=40831
--webserver_interface=127.14.106.193
--builtin_ntp_servers=127.14.106.212:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 20:09:13.557917 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:09:13.599180 18905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:13.599467 18905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:13.599625 18905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:13.600603 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:09:13.607259 18905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:13.607381 18905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:13.607533 18905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.193
W20240501 20:09:13.614984 15011 consensus_peers.cc:485] T bd92dbe7ec734430a5ab62db5b827849 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:09:13.617225 18905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.193:44871
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.106.193
--webserver_port=40831
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.14.106.254:34075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.193
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:13.618489 18905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:13.620191 18905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:13.632412 18910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:13.633571 18911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:13.634004 18913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:13.634922 18905 server_base.cc:1034] running on GCE node
I20240501 20:09:13.635522 18905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:13.636612 18905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:13.637933 18905 hybrid_clock.cc:648] HybridClock initialized: now 1714594153637864 us; error 65 us; skew 500 ppm
I20240501 20:09:13.638245 18905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:13.640465 18905 webserver.cc:460] Webserver started at http://127.14.106.193:40831/ using document root <none> and password file <none>
I20240501 20:09:13.641180 18905 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:13.641301 18905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:13.646713 18905 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
W20240501 20:09:13.649713 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:09:13.651626 18919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:13.652969 18905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:09:13.653115 18905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7decc3355b114404abddb7a10cc4358a"
format_stamp: "Formatted at 2024-05-01 20:08:24 on dist-test-slave-pg4x"
I20240501 20:09:13.653935 18905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:13.674561 18905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:13.675385 18905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:13.675637 18905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:13.676496 18905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:13.678463 18926 ts_tablet_manager.cc:522] Loading tablet metadata (0/12 complete)
I20240501 20:09:13.703460 18905 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240501 20:09:13.703567 18905 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.026s user 0.001s sys 0.001s
I20240501 20:09:13.703685 18905 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240501 20:09:13.707123 18926 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:13.733006 18926 log.cc:824] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:13.735366 18905 ts_tablet_manager.cc:590] Registered 12 tablets
I20240501 20:09:13.735487 18905 ts_tablet_manager.cc:569] Time spent register tablets: real 0.032s user 0.024s sys 0.008s
I20240501 20:09:13.786415 18926 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:13.787051 18926 tablet_bootstrap.cc:492] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:13.788543 18926 ts_tablet_manager.cc:1367] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.082s user 0.076s sys 0.002s
I20240501 20:09:13.788810 18905 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.193:44871
I20240501 20:09:13.788897 19033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.193:44871 every 8 connection(s)
I20240501 20:09:13.790270 18905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:09:13.799695 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 18905
I20240501 20:09:13.800133 18926 raft_consensus.cc:348] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:09:13.801465 18926 raft_consensus.cc:729] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.802440 18926 consensus_queue.cc:260] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:09:13.811107 19034 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34075
I20240501 20:09:13.811461 19034 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:13.812666 19034 heartbeater.cc:511] Master 127.14.106.254:34075 requested a full tablet report, sending...
I20240501 20:09:13.816956 14803 ts_manager.cc:194] Re-registered known tserver with Master: 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871)
I20240501 20:09:13.819877 18926 ts_tablet_manager.cc:1398] T 6513ee10256249b8976bfeb4cf0bdbb7 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.031s user 0.011s sys 0.009s
I20240501 20:09:13.820232 18926 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:13.822551 14803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.193:56829
W20240501 20:09:13.823626 15011 consensus_peers.cc:485] T 69d637373a3b4d2a9d807ccdb716569e P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 56: this message will repeat every 5th retry.
I20240501 20:09:13.826809 19034 heartbeater.cc:503] Master 127.14.106.254:34075 was elected leader, sending a full tablet report...
I20240501 20:09:13.862067 18926 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:13.862581 18926 tablet_bootstrap.cc:492] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:13.863587 18926 ts_tablet_manager.cc:1367] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.043s user 0.023s sys 0.019s
I20240501 20:09:13.864428 18926 raft_consensus.cc:348] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:09:13.864598 18926 raft_consensus.cc:729] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.864872 18926 consensus_queue.cc:260] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:09:13.865332 18926 ts_tablet_manager.cc:1398] T 3931f8694a4a45ee81497f1c0b263d4d P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:13.865582 18926 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:13.901788 18926 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:13.902273 18926 tablet_bootstrap.cc:492] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:13.903348 18926 ts_tablet_manager.cc:1367] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.038s user 0.029s sys 0.007s
I20240501 20:09:13.903978 18926 raft_consensus.cc:348] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:09:13.904168 18926 raft_consensus.cc:729] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.904430 18926 consensus_queue.cc:260] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:09:13.904930 18926 ts_tablet_manager.cc:1398] T 051f09e892c043269ed3e749c8f393fc P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:13.905448 18926 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:13.958576 18926 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:13.959264 18926 tablet_bootstrap.cc:492] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:13.960584 18926 ts_tablet_manager.cc:1367] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.055s user 0.042s sys 0.011s
I20240501 20:09:13.961485 18926 raft_consensus.cc:348] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:09:13.961814 18926 raft_consensus.cc:729] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.962224 18926 consensus_queue.cc:260] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:09:13.962970 18926 ts_tablet_manager.cc:1398] T bd92dbe7ec734430a5ab62db5b827849 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:13.963304 18926 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:14.019368 18926 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:14.020025 18926 tablet_bootstrap.cc:492] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:14.021446 18926 ts_tablet_manager.cc:1367] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.058s user 0.044s sys 0.011s
I20240501 20:09:14.022498 18926 raft_consensus.cc:348] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:09:14.022878 18926 raft_consensus.cc:729] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.023213 18926 consensus_queue.cc:260] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:09:14.023751 18926 ts_tablet_manager.cc:1398] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:09:14.024031 18926 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:14.068684 18926 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:14.069144 18926 tablet_bootstrap.cc:492] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:14.070088 18926 ts_tablet_manager.cc:1367] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.046s user 0.031s sys 0.012s
I20240501 20:09:14.070770 18926 raft_consensus.cc:348] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:09:14.070976 18926 raft_consensus.cc:729] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.071233 18926 consensus_queue.cc:260] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } attrs { replace: true } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "98d6b3fd85674334a24cd273bf525324" member_type: NON_VOTER last_known_addr { host: "127.14.106.198" port: 38031 } attrs { promote: true } }
I20240501 20:09:14.071874 18926 ts_tablet_manager.cc:1398] T c28b52cf6d9d4fa289de88ef97e08bee P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:09:14.072232 18926 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:14.138223 18926 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:14.138752 18926 tablet_bootstrap.cc:492] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:14.139744 18926 ts_tablet_manager.cc:1367] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.068s user 0.048s sys 0.010s
I20240501 20:09:14.140378 18926 raft_consensus.cc:348] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:09:14.140573 18926 raft_consensus.cc:729] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.140794 18926 consensus_queue.cc:260] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:09:14.141278 18926 ts_tablet_manager.cc:1398] T 69d637373a3b4d2a9d807ccdb716569e P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:14.141592 18926 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:14.198065 18926 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:14.198729 18926 tablet_bootstrap.cc:492] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:14.200018 18926 ts_tablet_manager.cc:1367] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.058s user 0.049s sys 0.008s
I20240501 20:09:14.200865 18926 raft_consensus.cc:348] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:09:14.201099 18926 raft_consensus.cc:729] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.201387 18926 consensus_queue.cc:260] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } }
I20240501 20:09:14.201987 18926 ts_tablet_manager.cc:1398] T dcc9afae97ec4d6fabefad1a319e913c P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:14.202270 18926 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:14.243970 18926 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:14.244560 18926 tablet_bootstrap.cc:492] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:14.245648 18926 ts_tablet_manager.cc:1367] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.043s user 0.032s sys 0.008s
I20240501 20:09:14.246256 18926 raft_consensus.cc:348] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:09:14.246443 18926 raft_consensus.cc:729] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.246680 18926 consensus_queue.cc:260] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } }
I20240501 20:09:14.247228 18926 ts_tablet_manager.cc:1398] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 20:09:14.247602 18926 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:14.303397 18926 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:14.304540 18926 tablet_bootstrap.cc:492] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:14.305907 18926 ts_tablet_manager.cc:1367] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.058s user 0.047s sys 0.007s
I20240501 20:09:14.306692 18926 raft_consensus.cc:348] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:09:14.306941 18926 raft_consensus.cc:729] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.307216 18926 consensus_queue.cc:260] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:09:14.307741 18926 ts_tablet_manager.cc:1398] T 2a672748de3e4e60bd950d6cffbcfd08 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:09:14.308036 18926 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:14.362671 18926 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:14.363361 18926 tablet_bootstrap.cc:492] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:14.364688 18926 ts_tablet_manager.cc:1367] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.057s user 0.035s sys 0.020s
I20240501 20:09:14.365378 18926 raft_consensus.cc:348] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:09:14.365684 18926 raft_consensus.cc:729] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.366022 18926 consensus_queue.cc:260] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } } peers { permanent_uuid: "4ed2ac58f951430a8c7e084f97d7a2a4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 34059 } }
I20240501 20:09:14.366688 18926 ts_tablet_manager.cc:1398] T 9c6758c002cc46f4bf960bff1196c457 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:09:14.367024 18926 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Bootstrap starting.
I20240501 20:09:14.388092 18989 raft_consensus.cc:3037] T 7e54c2e99947456988c55a7e5c5c4f53 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:14.423542 16814 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:14.428512 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:14.443171 16422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.448361 18926 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:14.449085 18926 tablet_bootstrap.cc:492] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Bootstrap complete.
I20240501 20:09:14.450615 18926 ts_tablet_manager.cc:1367] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Time spent bootstrapping tablet: real 0.084s user 0.057s sys 0.015s
I20240501 20:09:14.451741 18926 raft_consensus.cc:348] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:09:14.452114 18926 raft_consensus.cc:729] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7decc3355b114404abddb7a10cc4358a, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.452476 18926 consensus_queue.cc:260] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7decc3355b114404abddb7a10cc4358a" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 44871 } } peers { permanent_uuid: "2177fb70b1324acb81b2bea7be19955d" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 34095 } } peers { permanent_uuid: "e494ca2918594ba792c2d37d98b8c607" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 39333 } }
I20240501 20:09:14.453354 18989 raft_consensus.cc:3037] T 3e873682369f4d4f93dbe5fb3b369d1f P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:14.454182 18926 ts_tablet_manager.cc:1398] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:14.467682 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:14.498725 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:14.500732 18969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:14.503109 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:14.522254 15317 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:14.625932 18989 raft_consensus.cc:3037] T 95e65bcdb5174beb91a9edb54d20a102 P 7decc3355b114404abddb7a10cc4358a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:14.713937 16814 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:14.719311 15188 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:14.728953 18969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:14.741384 15057 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:14.745913 16422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.758675 16553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:14.769695 15317 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:14.813683 16684 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:14.987659 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 18905
I20240501 20:09:14.995491 19028 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:15.111547 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 18905
W20240501 20:09:15.136065 15141 connection.cc:535] client connection to 127.14.106.193:44871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:09:15.136632 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 14996
W20240501 20:09:15.138173 15141 consensus_peers.cc:485] T 95e65bcdb5174beb91a9edb54d20a102 P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:15.143246 15011 consensus_peers.cc:485] T 2a672748de3e4e60bd950d6cffbcfd08 P 4ed2ac58f951430a8c7e084f97d7a2a4 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:15.143378 15118 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:15.266711 15271 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:15.312752 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 14996
W20240501 20:09:15.326474 15271 consensus_peers.cc:485] T 7e54c2e99947456988c55a7e5c5c4f53 P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:15.334705 15141 consensus_peers.cc:485] T dcc9afae97ec4d6fabefad1a319e913c P 2177fb70b1324acb81b2bea7be19955d -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:15.342064 15271 connection.cc:535] server connection from 127.14.106.194:53737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:09:15.342388 15141 connection.cc:535] server connection from 127.14.106.194:44431 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:09:15.342757 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15126
I20240501 20:09:15.348433 15247 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:15.371472 15271 consensus_peers.cc:485] T 051f09e892c043269ed3e749c8f393fc P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:15.432701 15271 consensus_peers.cc:485] T 3e873682369f4d4f93dbe5fb3b369d1f P e494ca2918594ba792c2d37d98b8c607 -> Peer 7decc3355b114404abddb7a10cc4358a (127.14.106.193:44871): Couldn't send request to peer 7decc3355b114404abddb7a10cc4358a. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:44871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:15.515448 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15126
W20240501 20:09:15.541628 15274 consensus_peers.cc:485] T 9c6758c002cc46f4bf960bff1196c457 P e494ca2918594ba792c2d37d98b8c607 -> Peer 4ed2ac58f951430a8c7e084f97d7a2a4 (127.14.106.194:34059): Couldn't send request to peer 4ed2ac58f951430a8c7e084f97d7a2a4. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.194:34059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:15.544667 16637 connection.cc:535] server connection from 127.14.106.195:34353 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:09:15.545533 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15256
I20240501 20:09:15.551357 15376 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:15.719080 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15256
I20240501 20:09:15.751356 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 16357
I20240501 20:09:15.757095 16482 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:15.888495 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 16357
I20240501 20:09:15.912925 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 16491
I20240501 20:09:15.919006 16612 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:16.043567 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 16491
I20240501 20:09:16.066184 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 16621
I20240501 20:09:16.072069 16744 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:16.144397 16767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:38031: connect: Connection refused (error 111)
W20240501 20:09:16.145730 16767 consensus_peers.cc:485] T efbf0e49f78a4ff099786f47eedc3c8b P 0d9d84f0f3224d1a8fbaedd1021d730f -> Peer 98d6b3fd85674334a24cd273bf525324 (127.14.106.198:38031): Couldn't send request to peer 98d6b3fd85674334a24cd273bf525324. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:38031: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:16.189626 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 16621
I20240501 20:09:16.211288 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 16752
I20240501 20:09:16.217242 16873 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:16.347703 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 16752
I20240501 20:09:16.371079 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 14775
I20240501 20:09:16.372157 14836 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:16.536159 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 14775
2024-05-01T20:09:16Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (52961 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (52962 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240501 20:09:16.605402 14763 test_util.cc:268] Using random seed: 262369203
I20240501 20:09:16.607242 14763 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:09:16.607331 14763 ts_itest-base.cc:113] --------------
I20240501 20:09:16.607386 14763 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:09:16.607456 14763 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:09:16.607510 14763 ts_itest-base.cc:116] --------------
2024-05-01T20:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:09:16Z Disabled control of system clock
I20240501 20:09:16.631744 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:42629
--webserver_interface=127.14.106.254
--webserver_port=0
--builtin_ntp_servers=127.14.106.212:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.106.254:42629
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 20:09:16.948012 19185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:16.948330 19185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:16.948454 19185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:16.955763 19185 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:16.955915 19185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:09:16.955989 19185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:16.956079 19185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:09:16.956140 19185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:09:16.965201 19185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.106.254:42629
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:16.966559 19185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:16.968272 19185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:16.979348 19190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:16.980301 19191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:16.981161 19193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:16.981990 19185 server_base.cc:1034] running on GCE node
I20240501 20:09:16.982789 19185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:16.984169 19185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:16.985350 19185 hybrid_clock.cc:648] HybridClock initialized: now 1714594156985298 us; error 50 us; skew 500 ppm
I20240501 20:09:16.985711 19185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:16.988008 19185 webserver.cc:460] Webserver started at http://127.14.106.254:37297/ using document root <none> and password file <none>
I20240501 20:09:16.988736 19185 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:16.988852 19185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:16.989162 19185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:16.993505 19185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "221102bbcd56465f9ea89f2ba86229f1"
format_stamp: "Formatted at 2024-05-01 20:09:16 on dist-test-slave-pg4x"
I20240501 20:09:16.994288 19185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "221102bbcd56465f9ea89f2ba86229f1"
format_stamp: "Formatted at 2024-05-01 20:09:16 on dist-test-slave-pg4x"
I20240501 20:09:16.999575 19185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:09:17.002912 19199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:17.004171 19185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:09:17.004330 19185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
uuid: "221102bbcd56465f9ea89f2ba86229f1"
format_stamp: "Formatted at 2024-05-01 20:09:16 on dist-test-slave-pg4x"
I20240501 20:09:17.004525 19185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:17.025918 19185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:17.026768 19185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:17.027029 19185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:17.044569 19185 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.254:42629
I20240501 20:09:17.044610 19250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.254:42629 every 8 connection(s)
I20240501 20:09:17.045897 19185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:09:17.047966 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 19185
I20240501 20:09:17.048209 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:09:17.050367 19251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.062288 19251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1: Bootstrap starting.
I20240501 20:09:17.065086 19251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.066018 19251 log.cc:824] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:17.068379 19251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1: No bootstrap required, opened a new log
I20240501 20:09:17.074249 19251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221102bbcd56465f9ea89f2ba86229f1" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 42629 } }
I20240501 20:09:17.074599 19251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.074731 19251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221102bbcd56465f9ea89f2ba86229f1, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.075275 19251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221102bbcd56465f9ea89f2ba86229f1" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 42629 } }
I20240501 20:09:17.075462 19251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:17.075570 19251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:17.075726 19251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.077769 19251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221102bbcd56465f9ea89f2ba86229f1" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 42629 } }
I20240501 20:09:17.078258 19251 leader_election.cc:302] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221102bbcd56465f9ea89f2ba86229f1; no voters:
I20240501 20:09:17.078858 19251 leader_election.cc:288] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:17.078975 19256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:17.079592 19256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [term 1 LEADER]: Becoming Leader. State: Replica: 221102bbcd56465f9ea89f2ba86229f1, State: Running, Role: LEADER
I20240501 20:09:17.080137 19256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221102bbcd56465f9ea89f2ba86229f1" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 42629 } }
I20240501 20:09:17.080741 19251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:17.083196 19258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221102bbcd56465f9ea89f2ba86229f1. Latest consensus state: current_term: 1 leader_uuid: "221102bbcd56465f9ea89f2ba86229f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221102bbcd56465f9ea89f2ba86229f1" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 42629 } } }
I20240501 20:09:17.083333 19257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221102bbcd56465f9ea89f2ba86229f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221102bbcd56465f9ea89f2ba86229f1" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 42629 } } }
I20240501 20:09:17.083643 19257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:17.083652 19258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:17.090574 19265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:17.095518 19265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:17.103294 19265 catalog_manager.cc:1347] Generated new cluster ID: 62203b65bfa9479fa13c22d97e9672f9
I20240501 20:09:17.103416 19265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:17.112901 19265 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:17.114140 19265 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:17.124750 19265 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 221102bbcd56465f9ea89f2ba86229f1: Generated new TSK 0
I20240501 20:09:17.125645 19265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:17.140774 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.193:0
--local_ip_for_outbound_sockets=127.14.106.193
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--builtin_ntp_servers=127.14.106.212:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:09:17.473870 19275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:17.474156 19275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:17.474282 19275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:17.481710 19275 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:17.481853 19275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:17.482044 19275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.193
I20240501 20:09:17.491063 19275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.193
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:17.492333 19275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:17.493992 19275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:17.506065 19281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:17.506899 19280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:17.507695 19283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:17.508576 19275 server_base.cc:1034] running on GCE node
I20240501 20:09:17.509169 19275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:17.510092 19275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:17.511319 19275 hybrid_clock.cc:648] HybridClock initialized: now 1714594157511266 us; error 34 us; skew 500 ppm
I20240501 20:09:17.511628 19275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:17.513718 19275 webserver.cc:460] Webserver started at http://127.14.106.193:38235/ using document root <none> and password file <none>
I20240501 20:09:17.514411 19275 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:17.514520 19275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:17.514844 19275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:17.517246 19275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4a277d5d99b940109476f70239050214"
format_stamp: "Formatted at 2024-05-01 20:09:17 on dist-test-slave-pg4x"
I20240501 20:09:17.517948 19275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4a277d5d99b940109476f70239050214"
format_stamp: "Formatted at 2024-05-01 20:09:17 on dist-test-slave-pg4x"
I20240501 20:09:17.522729 19275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240501 20:09:17.525960 19289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:17.527200 19275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:17.527359 19275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4a277d5d99b940109476f70239050214"
format_stamp: "Formatted at 2024-05-01 20:09:17 on dist-test-slave-pg4x"
I20240501 20:09:17.527542 19275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:17.551164 19275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:17.552027 19275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:17.552294 19275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:17.553241 19275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:17.554708 19275 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:17.554806 19275 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:17.554921 19275 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:17.554998 19275 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:17.587369 19275 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.193:36073
I20240501 20:09:17.587451 19401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.193:36073 every 8 connection(s)
I20240501 20:09:17.588608 19275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:09:17.590101 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 19275
I20240501 20:09:17.590281 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:09:17.594765 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.194:0
--local_ip_for_outbound_sockets=127.14.106.194
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--builtin_ntp_servers=127.14.106.212:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:17.603914 19402 heartbeater.cc:347] Connected to a master server at 127.14.106.254:42629
I20240501 20:09:17.604192 19402 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:17.604880 19402 heartbeater.cc:511] Master 127.14.106.254:42629 requested a full tablet report, sending...
I20240501 20:09:17.606796 19216 ts_manager.cc:194] Registered new tserver with Master: 4a277d5d99b940109476f70239050214 (127.14.106.193:36073)
I20240501 20:09:17.608268 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.193:36015
W20240501 20:09:17.914011 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:17.914302 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:17.914443 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:17.921713 19406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:17.921877 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:17.922031 19406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.194
I20240501 20:09:17.930945 19406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.194
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:17.932251 19406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:17.934135 19406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:17.946733 19411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:17.947569 19414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:17.948289 19412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:17.949246 19406 server_base.cc:1034] running on GCE node
I20240501 20:09:17.949816 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:17.950789 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:17.952003 19406 hybrid_clock.cc:648] HybridClock initialized: now 1714594157951955 us; error 38 us; skew 500 ppm
I20240501 20:09:17.952327 19406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:17.954452 19406 webserver.cc:460] Webserver started at http://127.14.106.194:35835/ using document root <none> and password file <none>
I20240501 20:09:17.955190 19406 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:17.955292 19406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:17.955603 19406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:17.958072 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "57937826d72b4b4aa7b98b0db7c13c10"
format_stamp: "Formatted at 2024-05-01 20:09:17 on dist-test-slave-pg4x"
I20240501 20:09:17.958812 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "57937826d72b4b4aa7b98b0db7c13c10"
format_stamp: "Formatted at 2024-05-01 20:09:17 on dist-test-slave-pg4x"
I20240501 20:09:17.963639 19406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 20:09:17.966885 19420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:17.968127 19406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:09:17.968264 19406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "57937826d72b4b4aa7b98b0db7c13c10"
format_stamp: "Formatted at 2024-05-01 20:09:17 on dist-test-slave-pg4x"
I20240501 20:09:17.968466 19406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:17.981662 19406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:17.982404 19406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:17.982674 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:17.983529 19406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:17.984875 19406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:17.984961 19406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:17.985065 19406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:17.985132 19406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:18.015242 19406 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.194:34699
I20240501 20:09:18.015304 19532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.194:34699 every 8 connection(s)
I20240501 20:09:18.016490 19406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:09:18.022743 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 19406
I20240501 20:09:18.022946 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:09:18.025776 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.195:0
--local_ip_for_outbound_sockets=127.14.106.195
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--builtin_ntp_servers=127.14.106.212:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:18.028723 19533 heartbeater.cc:347] Connected to a master server at 127.14.106.254:42629
I20240501 20:09:18.028997 19533 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:18.029644 19533 heartbeater.cc:511] Master 127.14.106.254:42629 requested a full tablet report, sending...
I20240501 20:09:18.030946 19216 ts_manager.cc:194] Registered new tserver with Master: 57937826d72b4b4aa7b98b0db7c13c10 (127.14.106.194:34699)
I20240501 20:09:18.031699 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.194:40981
W20240501 20:09:18.346871 19536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:18.347160 19536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:18.347303 19536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:18.354451 19536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:18.354595 19536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:18.354761 19536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.195
I20240501 20:09:18.363281 19536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.195
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:18.364509 19536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:18.366155 19536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:18.378227 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:18.379052 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:18.379889 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:18.380676 19536 server_base.cc:1034] running on GCE node
I20240501 20:09:18.381307 19536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:18.382223 19536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:18.383421 19536 hybrid_clock.cc:648] HybridClock initialized: now 1714594158383364 us; error 55 us; skew 500 ppm
I20240501 20:09:18.383710 19536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:18.385778 19536 webserver.cc:460] Webserver started at http://127.14.106.195:41253/ using document root <none> and password file <none>
I20240501 20:09:18.386437 19536 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:18.386544 19536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:18.386898 19536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:18.389439 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "615960a568d24456b6a02c8b3995e457"
format_stamp: "Formatted at 2024-05-01 20:09:18 on dist-test-slave-pg4x"
I20240501 20:09:18.390141 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "615960a568d24456b6a02c8b3995e457"
format_stamp: "Formatted at 2024-05-01 20:09:18 on dist-test-slave-pg4x"
I20240501 20:09:18.395128 19536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:09:18.398298 19550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:18.399607 19536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:18.399752 19536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "615960a568d24456b6a02c8b3995e457"
format_stamp: "Formatted at 2024-05-01 20:09:18 on dist-test-slave-pg4x"
I20240501 20:09:18.399933 19536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:18.418083 19536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:18.418813 19536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:18.419171 19536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:18.420084 19536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:18.421481 19536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:18.421569 19536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:18.421676 19536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:18.421744 19536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:18.450527 19536 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.195:36435
I20240501 20:09:18.450569 19662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.195:36435 every 8 connection(s)
I20240501 20:09:18.451799 19536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:09:18.453632 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 19536
I20240501 20:09:18.453788 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:09:18.456689 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:0
--local_ip_for_outbound_sockets=127.14.106.196
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--builtin_ntp_servers=127.14.106.212:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:18.464982 19663 heartbeater.cc:347] Connected to a master server at 127.14.106.254:42629
I20240501 20:09:18.465245 19663 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:18.465960 19663 heartbeater.cc:511] Master 127.14.106.254:42629 requested a full tablet report, sending...
I20240501 20:09:18.467315 19216 ts_manager.cc:194] Registered new tserver with Master: 615960a568d24456b6a02c8b3995e457 (127.14.106.195:36435)
I20240501 20:09:18.468106 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.195:37713
I20240501 20:09:18.610945 19402 heartbeater.cc:503] Master 127.14.106.254:42629 was elected leader, sending a full tablet report...
W20240501 20:09:18.780377 19666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:18.780658 19666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:18.780793 19666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:18.788275 19666 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:18.788409 19666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:18.788559 19666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:09:18.797483 19666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:18.798729 19666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:18.800417 19666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:18.812532 19671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:18.813403 19674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:18.814209 19672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:18.815117 19666 server_base.cc:1034] running on GCE node
I20240501 20:09:18.815701 19666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:18.816583 19666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:18.817786 19666 hybrid_clock.cc:648] HybridClock initialized: now 1714594158817745 us; error 38 us; skew 500 ppm
I20240501 20:09:18.818101 19666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:18.824057 19666 webserver.cc:460] Webserver started at http://127.14.106.196:45799/ using document root <none> and password file <none>
I20240501 20:09:18.824786 19666 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:18.824908 19666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:18.825223 19666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:18.827901 19666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c0cb2864016c406595597f20b7ee9c4f"
format_stamp: "Formatted at 2024-05-01 20:09:18 on dist-test-slave-pg4x"
I20240501 20:09:18.828601 19666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c0cb2864016c406595597f20b7ee9c4f"
format_stamp: "Formatted at 2024-05-01 20:09:18 on dist-test-slave-pg4x"
I20240501 20:09:18.833480 19666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:09:18.836639 19680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:18.837863 19666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:18.838012 19666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c0cb2864016c406595597f20b7ee9c4f"
format_stamp: "Formatted at 2024-05-01 20:09:18 on dist-test-slave-pg4x"
I20240501 20:09:18.838201 19666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:18.867705 19666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:18.868595 19666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:18.868868 19666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:18.869849 19666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:18.871378 19666 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:18.871474 19666 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:18.871580 19666 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:18.871652 19666 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:18.909276 19666 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:35299
I20240501 20:09:18.909332 19792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:35299 every 8 connection(s)
I20240501 20:09:18.911011 19666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:09:18.915223 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 19666
I20240501 20:09:18.915429 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:09:18.919010 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:0
--local_ip_for_outbound_sockets=127.14.106.197
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--builtin_ntp_servers=127.14.106.212:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:18.925416 19793 heartbeater.cc:347] Connected to a master server at 127.14.106.254:42629
I20240501 20:09:18.925797 19793 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:18.926893 19793 heartbeater.cc:511] Master 127.14.106.254:42629 requested a full tablet report, sending...
I20240501 20:09:18.928500 19216 ts_manager.cc:194] Registered new tserver with Master: c0cb2864016c406595597f20b7ee9c4f (127.14.106.196:35299)
I20240501 20:09:18.929617 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:58047
I20240501 20:09:19.034433 19533 heartbeater.cc:503] Master 127.14.106.254:42629 was elected leader, sending a full tablet report...
W20240501 20:09:19.248572 19796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:19.248852 19796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:19.248939 19796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:19.256044 19796 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:19.256157 19796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:19.256271 19796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:09:19.264783 19796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:42629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:19.266016 19796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:19.267661 19796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:19.279724 19801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:19.280594 19802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:19.281412 19804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:19.282323 19796 server_base.cc:1034] running on GCE node
I20240501 20:09:19.282967 19796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:19.283854 19796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:19.285053 19796 hybrid_clock.cc:648] HybridClock initialized: now 1714594159285006 us; error 42 us; skew 500 ppm
I20240501 20:09:19.285343 19796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:19.292376 19796 webserver.cc:460] Webserver started at http://127.14.106.197:36189/ using document root <none> and password file <none>
I20240501 20:09:19.293116 19796 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:19.293229 19796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:19.293521 19796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:19.296078 19796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "88ddd5bfd0e841eda8523de46bd5ce2c"
format_stamp: "Formatted at 2024-05-01 20:09:19 on dist-test-slave-pg4x"
I20240501 20:09:19.296777 19796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "88ddd5bfd0e841eda8523de46bd5ce2c"
format_stamp: "Formatted at 2024-05-01 20:09:19 on dist-test-slave-pg4x"
I20240501 20:09:19.301546 19796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240501 20:09:19.304742 19810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:19.306058 19796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:19.306205 19796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "88ddd5bfd0e841eda8523de46bd5ce2c"
format_stamp: "Formatted at 2024-05-01 20:09:19 on dist-test-slave-pg4x"
I20240501 20:09:19.306386 19796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:19.335541 19796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:19.336414 19796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:19.336686 19796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:19.337646 19796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:19.339130 19796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:19.339231 19796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:19.339340 19796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:19.339411 19796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:19.370747 19796 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:40343
I20240501 20:09:19.370821 19922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:40343 every 8 connection(s)
I20240501 20:09:19.372061 19796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:19.378727 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 19796
I20240501 20:09:19.378911 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:09:19.384099 19923 heartbeater.cc:347] Connected to a master server at 127.14.106.254:42629
I20240501 20:09:19.384338 19923 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:19.384948 19923 heartbeater.cc:511] Master 127.14.106.254:42629 requested a full tablet report, sending...
I20240501 20:09:19.386034 19216 ts_manager.cc:194] Registered new tserver with Master: 88ddd5bfd0e841eda8523de46bd5ce2c (127.14.106.197:40343)
I20240501 20:09:19.386775 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:35215
I20240501 20:09:19.393929 14763 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:09:19.409248 19216 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38816:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:09:19.444334 19337 tablet_service.cc:1467] Processing CreateTablet for tablet e259d8877bbc4a8dba45927403427fde (DEFAULT_TABLE table=TestTable [id=3d86c4c962c54736b678a10bdab534a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.444823 19728 tablet_service.cc:1467] Processing CreateTablet for tablet e259d8877bbc4a8dba45927403427fde (DEFAULT_TABLE table=TestTable [id=3d86c4c962c54736b678a10bdab534a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.444680 19598 tablet_service.cc:1467] Processing CreateTablet for tablet e259d8877bbc4a8dba45927403427fde (DEFAULT_TABLE table=TestTable [id=3d86c4c962c54736b678a10bdab534a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.445566 19337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259d8877bbc4a8dba45927403427fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.446015 19728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259d8877bbc4a8dba45927403427fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.446031 19598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259d8877bbc4a8dba45927403427fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.463944 19941 tablet_bootstrap.cc:492] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214: Bootstrap starting.
I20240501 20:09:19.464744 19942 tablet_bootstrap.cc:492] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f: Bootstrap starting.
I20240501 20:09:19.465528 19943 tablet_bootstrap.cc:492] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457: Bootstrap starting.
I20240501 20:09:19.468914 19941 tablet_bootstrap.cc:654] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:19.469369 19943 tablet_bootstrap.cc:654] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:19.469527 19942 tablet_bootstrap.cc:654] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:19.470221 19941 log.cc:824] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:19.470548 19943 log.cc:824] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:19.470706 19942 log.cc:824] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:19.470551 19663 heartbeater.cc:503] Master 127.14.106.254:42629 was elected leader, sending a full tablet report...
I20240501 20:09:19.480329 19941 tablet_bootstrap.cc:492] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214: No bootstrap required, opened a new log
I20240501 20:09:19.480530 19943 tablet_bootstrap.cc:492] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457: No bootstrap required, opened a new log
I20240501 20:09:19.480659 19941 ts_tablet_manager.cc:1367] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20240501 20:09:19.480870 19943 ts_tablet_manager.cc:1367] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240501 20:09:19.482183 19942 tablet_bootstrap.cc:492] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f: No bootstrap required, opened a new log
I20240501 20:09:19.482445 19942 ts_tablet_manager.cc:1367] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20240501 20:09:19.488714 19942 raft_consensus.cc:348] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.489075 19942 raft_consensus.cc:374] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:19.489197 19942 raft_consensus.cc:729] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cb2864016c406595597f20b7ee9c4f, State: Initialized, Role: FOLLOWER
I20240501 20:09:19.489804 19942 consensus_queue.cc:260] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.489706 19941 raft_consensus.cc:348] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.490151 19941 raft_consensus.cc:374] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:19.490286 19941 raft_consensus.cc:729] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a277d5d99b940109476f70239050214, State: Initialized, Role: FOLLOWER
I20240501 20:09:19.490087 19943 raft_consensus.cc:348] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.490547 19943 raft_consensus.cc:374] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:19.490972 19793 heartbeater.cc:503] Master 127.14.106.254:42629 was elected leader, sending a full tablet report...
I20240501 20:09:19.490989 19941 consensus_queue.cc:260] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.491313 19942 ts_tablet_manager.cc:1398] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 20:09:19.491520 19943 raft_consensus.cc:729] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615960a568d24456b6a02c8b3995e457, State: Initialized, Role: FOLLOWER
I20240501 20:09:19.492292 19943 consensus_queue.cc:260] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.492585 19941 ts_tablet_manager.cc:1398] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20240501 20:09:19.495246 19943 ts_tablet_manager.cc:1398] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20240501 20:09:19.495808 19949 raft_consensus.cc:482] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:19.496106 19949 raft_consensus.cc:504] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.498296 19949 leader_election.cc:288] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a277d5d99b940109476f70239050214 (127.14.106.193:36073), c0cb2864016c406595597f20b7ee9c4f (127.14.106.196:35299)
I20240501 20:09:19.506377 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259d8877bbc4a8dba45927403427fde" candidate_uuid: "615960a568d24456b6a02c8b3995e457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a277d5d99b940109476f70239050214" is_pre_election: true
I20240501 20:09:19.506896 19357 raft_consensus.cc:2454] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615960a568d24456b6a02c8b3995e457 in term 0.
I20240501 20:09:19.507395 19748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259d8877bbc4a8dba45927403427fde" candidate_uuid: "615960a568d24456b6a02c8b3995e457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cb2864016c406595597f20b7ee9c4f" is_pre_election: true
I20240501 20:09:19.507529 19551 leader_election.cc:302] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a277d5d99b940109476f70239050214, 615960a568d24456b6a02c8b3995e457; no voters:
I20240501 20:09:19.507900 19748 raft_consensus.cc:2454] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615960a568d24456b6a02c8b3995e457 in term 0.
I20240501 20:09:19.507920 19950 raft_consensus.cc:2781] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:19.508044 19950 raft_consensus.cc:482] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:19.508322 19950 raft_consensus.cc:3037] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:19.510607 19950 raft_consensus.cc:504] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.511428 19950 leader_election.cc:288] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [CANDIDATE]: Term 1 election: Requested vote from peers 4a277d5d99b940109476f70239050214 (127.14.106.193:36073), c0cb2864016c406595597f20b7ee9c4f (127.14.106.196:35299)
I20240501 20:09:19.511780 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259d8877bbc4a8dba45927403427fde" candidate_uuid: "615960a568d24456b6a02c8b3995e457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a277d5d99b940109476f70239050214"
I20240501 20:09:19.511953 19357 raft_consensus.cc:3037] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:19.512053 19748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259d8877bbc4a8dba45927403427fde" candidate_uuid: "615960a568d24456b6a02c8b3995e457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cb2864016c406595597f20b7ee9c4f"
I20240501 20:09:19.512245 19748 raft_consensus.cc:3037] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:19.515133 19357 raft_consensus.cc:2454] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615960a568d24456b6a02c8b3995e457 in term 1.
I20240501 20:09:19.515233 19748 raft_consensus.cc:2454] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615960a568d24456b6a02c8b3995e457 in term 1.
I20240501 20:09:19.515558 19551 leader_election.cc:302] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a277d5d99b940109476f70239050214, 615960a568d24456b6a02c8b3995e457; no voters:
I20240501 20:09:19.515795 19950 raft_consensus.cc:2781] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:19.516057 19950 raft_consensus.cc:686] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [term 1 LEADER]: Becoming Leader. State: Replica: 615960a568d24456b6a02c8b3995e457, State: Running, Role: LEADER
I20240501 20:09:19.516552 19950 consensus_queue.cc:237] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } }
I20240501 20:09:19.520633 19214 catalog_manager.cc:5484] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 reported cstate change: term changed from 0 to 1, leader changed from <none> to 615960a568d24456b6a02c8b3995e457 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "615960a568d24456b6a02c8b3995e457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 } } peers { permanent_uuid: "615960a568d24456b6a02c8b3995e457" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 36435 } } peers { permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 } } }
I20240501 20:09:19.543757 14763 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:09:19.546489 14763 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4a277d5d99b940109476f70239050214 to finish bootstrapping
I20240501 20:09:19.555455 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 57937826d72b4b4aa7b98b0db7c13c10 to finish bootstrapping
I20240501 20:09:19.563638 14763 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 615960a568d24456b6a02c8b3995e457 to finish bootstrapping
I20240501 20:09:19.570302 14763 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c0cb2864016c406595597f20b7ee9c4f to finish bootstrapping
I20240501 20:09:19.577481 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 88ddd5bfd0e841eda8523de46bd5ce2c to finish bootstrapping
W20240501 20:09:19.591437 19403 tablet.cc:2369] T e259d8877bbc4a8dba45927403427fde P 4a277d5d99b940109476f70239050214: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:19.663290 19794 tablet.cc:2369] T e259d8877bbc4a8dba45927403427fde P c0cb2864016c406595597f20b7ee9c4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:19.704102 19664 tablet.cc:2369] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:20.035887 19337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.036460 19598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:20.038478 19468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.039812 19728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.040364 19858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.073505 19988 consensus_queue.cc:1035] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a277d5d99b940109476f70239050214" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:20.104518 19988 consensus_queue.cc:1035] T e259d8877bbc4a8dba45927403427fde P 615960a568d24456b6a02c8b3995e457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0cb2864016c406595597f20b7ee9c4f" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:20.228015 19337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.229007 19468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.242316 19858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.244738 19598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:20.267412 19728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.388862 19923 heartbeater.cc:503] Master 127.14.106.254:42629 was elected leader, sending a full tablet report...
I20240501 20:09:20.850445 19468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.850661 19337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.867619 19598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:20.872831 19858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:20.873571 19728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.033195 19468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.033685 19337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.041785 19598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:21.069643 19728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.074915 19858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.219655 19468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.221354 19337 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.229590 19598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:21.241395 19858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.264721 19728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:21.406720 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 19275
I20240501 20:09:21.408206 19396 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:21.507138 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 19275
I20240501 20:09:21.525794 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 19406
I20240501 20:09:21.526899 19528 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:21.617641 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 19406
I20240501 20:09:21.635125 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 19536
I20240501 20:09:21.636157 19657 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:21.731209 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 19536
I20240501 20:09:21.749976 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 19666
I20240501 20:09:21.751008 19787 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:21.846210 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 19666
I20240501 20:09:21.864041 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 19796
I20240501 20:09:21.865082 19917 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:21.955654 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 19796
I20240501 20:09:21.972858 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 19185
I20240501 20:09:21.973968 19246 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:22.086555 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 19185
2024-05-01T20:09:22Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (5524 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5524 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240501 20:09:22.129487 14763 test_util.cc:268] Using random seed: 267893301
I20240501 20:09:22.131350 14763 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:09:22.131415 14763 ts_itest-base.cc:113] --------------
I20240501 20:09:22.131456 14763 ts_itest-base.cc:114] 7 tablet servers
I20240501 20:09:22.131492 14763 ts_itest-base.cc:115] 2 replicas per TS
I20240501 20:09:22.131534 14763 ts_itest-base.cc:116] --------------
2024-05-01T20:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:09:22Z Disabled control of system clock
I20240501 20:09:22.155953 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:36189
--webserver_interface=127.14.106.254
--webserver_port=0
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.106.254:36189
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 20:09:22.471386 20148 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:09:22.471691 20148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:22.471768 20148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:22.471841 20148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:22.479043 20148 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:22.479153 20148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:09:22.479211 20148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:22.479259 20148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:09:22.479315 20148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:09:22.488176 20148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.106.254:36189
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:22.489429 20148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:22.491238 20148 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:22.502164 20153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.503015 20154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.503823 20156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:22.504609 20148 server_base.cc:1034] running on GCE node
I20240501 20:09:22.505373 20148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:22.506906 20148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:22.508142 20148 hybrid_clock.cc:648] HybridClock initialized: now 1714594162508074 us; error 44 us; skew 500 ppm
I20240501 20:09:22.508481 20148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:22.510839 20148 webserver.cc:460] Webserver started at http://127.14.106.254:46565/ using document root <none> and password file <none>
I20240501 20:09:22.511536 20148 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:22.511670 20148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:22.511969 20148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:22.514529 20148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a92b9fd7e9164dc18d7bb7cea432762a"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-pg4x"
I20240501 20:09:22.515316 20148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a92b9fd7e9164dc18d7bb7cea432762a"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-pg4x"
I20240501 20:09:22.520337 20148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:09:22.523595 20162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:22.524889 20148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:22.525053 20148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a92b9fd7e9164dc18d7bb7cea432762a"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-pg4x"
I20240501 20:09:22.525247 20148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:22.543754 20148 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:22.544576 20148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:22.544870 20148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:22.562652 20148 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.254:36189
I20240501 20:09:22.562698 20213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.254:36189 every 8 connection(s)
I20240501 20:09:22.564008 20148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:09:22.567723 20214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.572733 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 20148
I20240501 20:09:22.572976 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:09:22.577514 20214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a: Bootstrap starting.
I20240501 20:09:22.580943 20214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.581948 20214 log.cc:824] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:22.584995 20214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a: No bootstrap required, opened a new log
I20240501 20:09:22.591650 20214 raft_consensus.cc:348] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92b9fd7e9164dc18d7bb7cea432762a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 36189 } }
I20240501 20:09:22.592026 20214 raft_consensus.cc:374] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.592135 20214 raft_consensus.cc:729] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a92b9fd7e9164dc18d7bb7cea432762a, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.592988 20214 consensus_queue.cc:260] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92b9fd7e9164dc18d7bb7cea432762a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 36189 } }
I20240501 20:09:22.593248 20214 raft_consensus.cc:388] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:22.593410 20214 raft_consensus.cc:482] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:22.593621 20214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.596172 20214 raft_consensus.cc:504] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92b9fd7e9164dc18d7bb7cea432762a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 36189 } }
I20240501 20:09:22.596639 20214 leader_election.cc:302] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a92b9fd7e9164dc18d7bb7cea432762a; no voters:
I20240501 20:09:22.597193 20214 leader_election.cc:288] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:22.597322 20219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:22.597965 20219 raft_consensus.cc:686] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [term 1 LEADER]: Becoming Leader. State: Replica: a92b9fd7e9164dc18d7bb7cea432762a, State: Running, Role: LEADER
I20240501 20:09:22.598507 20219 consensus_queue.cc:237] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92b9fd7e9164dc18d7bb7cea432762a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 36189 } }
I20240501 20:09:22.599325 20214 sys_catalog.cc:564] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:22.601615 20221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a92b9fd7e9164dc18d7bb7cea432762a. Latest consensus state: current_term: 1 leader_uuid: "a92b9fd7e9164dc18d7bb7cea432762a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92b9fd7e9164dc18d7bb7cea432762a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 36189 } } }
I20240501 20:09:22.602023 20221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:22.601635 20220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a92b9fd7e9164dc18d7bb7cea432762a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92b9fd7e9164dc18d7bb7cea432762a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 36189 } } }
I20240501 20:09:22.602327 20220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:22.606750 20225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:22.611647 20225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:22.619938 20225 catalog_manager.cc:1347] Generated new cluster ID: 96c2dff57d5c463cb03427538b6d6337
I20240501 20:09:22.620047 20225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:22.656316 20225 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:22.657486 20225 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:22.668052 20225 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a92b9fd7e9164dc18d7bb7cea432762a: Generated new TSK 0
I20240501 20:09:22.668828 20225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:22.680877 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.193:0
--local_ip_for_outbound_sockets=127.14.106.193
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:09:23.008999 20238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:23.009275 20238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:23.009400 20238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:23.016731 20238 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:23.016845 20238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:23.017000 20238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.193
I20240501 20:09:23.025946 20238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.193
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:23.027427 20238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:23.029192 20238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:23.042332 20244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.043200 20243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.043965 20246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:23.044939 20238 server_base.cc:1034] running on GCE node
I20240501 20:09:23.045529 20238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:23.046447 20238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:23.047650 20238 hybrid_clock.cc:648] HybridClock initialized: now 1714594163047592 us; error 45 us; skew 500 ppm
I20240501 20:09:23.047968 20238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:23.050166 20238 webserver.cc:460] Webserver started at http://127.14.106.193:34079/ using document root <none> and password file <none>
I20240501 20:09:23.050927 20238 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:23.051043 20238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:23.051347 20238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:23.054188 20238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4f8d9458da0b406db99095564f9cf2ed"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.054977 20238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4f8d9458da0b406db99095564f9cf2ed"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.060007 20238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20240501 20:09:23.063180 20252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.064528 20238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:09:23.064685 20238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4f8d9458da0b406db99095564f9cf2ed"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.064968 20238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.080853 20238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:23.081703 20238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.081990 20238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.082931 20238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.084319 20238 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.084424 20238 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.084535 20238 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.084609 20238 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.115581 20238 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.193:36237
I20240501 20:09:23.115662 20364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.193:36237 every 8 connection(s)
I20240501 20:09:23.116870 20238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:09:23.118522 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 20238
I20240501 20:09:23.118775 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:09:23.122535 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.194:0
--local_ip_for_outbound_sockets=127.14.106.194
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:23.131527 20365 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:23.131831 20365 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:23.132620 20365 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:23.135046 20179 ts_manager.cc:194] Registered new tserver with Master: 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:23.137430 20179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.193:42575
W20240501 20:09:23.444880 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:23.445151 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:23.445237 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:23.452469 20369 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:23.452587 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:23.452702 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.194
I20240501 20:09:23.461460 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.194
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:23.462707 20369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:23.464340 20369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:09:23.477329 20369 server_base.cc:1034] running on GCE node
W20240501 20:09:23.478138 20377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.478909 20375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.478956 20374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:23.479650 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:23.480783 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:23.481984 20369 hybrid_clock.cc:648] HybridClock initialized: now 1714594163481931 us; error 54 us; skew 500 ppm
I20240501 20:09:23.482297 20369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:23.484536 20369 webserver.cc:460] Webserver started at http://127.14.106.194:41353/ using document root <none> and password file <none>
I20240501 20:09:23.485256 20369 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:23.485376 20369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:23.485690 20369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:23.488219 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "28b7e229ab3d4f97a09ca51ac037b008"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.488927 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "28b7e229ab3d4f97a09ca51ac037b008"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.493765 20369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:09:23.496912 20383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.498122 20369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:23.498270 20369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "28b7e229ab3d4f97a09ca51ac037b008"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.498456 20369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.525768 20369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:23.526721 20369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.527029 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.528023 20369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.529644 20369 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.529750 20369 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.529888 20369 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.529966 20369 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.561409 20369 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.194:39937
I20240501 20:09:23.561499 20495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.194:39937 every 8 connection(s)
I20240501 20:09:23.563747 20369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:09:23.572005 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 20369
I20240501 20:09:23.572175 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:09:23.575474 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.195:0
--local_ip_for_outbound_sockets=127.14.106.195
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:23.579150 20496 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:23.579455 20496 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:23.580135 20496 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:23.581348 20179 ts_manager.cc:194] Registered new tserver with Master: 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:23.582098 20179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.194:59149
W20240501 20:09:23.903831 20499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:23.904119 20499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:23.904208 20499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:23.911832 20499 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:23.911957 20499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:23.912075 20499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.195
I20240501 20:09:23.921048 20499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.195
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:23.922331 20499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:23.924036 20499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:23.936414 20504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.937215 20507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.938109 20505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:23.938994 20499 server_base.cc:1034] running on GCE node
I20240501 20:09:23.939584 20499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:23.940470 20499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:23.941668 20499 hybrid_clock.cc:648] HybridClock initialized: now 1714594163941632 us; error 29 us; skew 500 ppm
I20240501 20:09:23.941998 20499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:23.944118 20499 webserver.cc:460] Webserver started at http://127.14.106.195:45951/ using document root <none> and password file <none>
I20240501 20:09:23.944792 20499 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:23.944919 20499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:23.945205 20499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:23.947724 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2305b26386ac446da2436463b536d296"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.948415 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2305b26386ac446da2436463b536d296"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.953189 20499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:09:23.956405 20513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.957594 20499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:23.957743 20499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2305b26386ac446da2436463b536d296"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-pg4x"
I20240501 20:09:23.957933 20499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.969723 20499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:23.970499 20499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.970793 20499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.971640 20499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.972981 20499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.973078 20499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.973191 20499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.973264 20499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.004736 20499 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.195:35301
I20240501 20:09:24.004787 20625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.195:35301 every 8 connection(s)
I20240501 20:09:24.006006 20499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:09:24.006712 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 20499
I20240501 20:09:24.006959 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:09:24.010233 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:0
--local_ip_for_outbound_sockets=127.14.106.196
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:24.020915 20626 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:24.021214 20626 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:24.021940 20626 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:24.023363 20179 ts_manager.cc:194] Registered new tserver with Master: 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:24.024140 20179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.195:38605
I20240501 20:09:24.140440 20365 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
W20240501 20:09:24.334970 20629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:24.335251 20629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:24.335353 20629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:24.342639 20629 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:24.342773 20629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:24.342897 20629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:09:24.351609 20629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:24.352865 20629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:24.354513 20629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:24.366704 20634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:24.367575 20637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:24.368324 20635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:24.369259 20629 server_base.cc:1034] running on GCE node
I20240501 20:09:24.369871 20629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:24.370820 20629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:24.372020 20629 hybrid_clock.cc:648] HybridClock initialized: now 1714594164371975 us; error 48 us; skew 500 ppm
I20240501 20:09:24.372323 20629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:24.380040 20629 webserver.cc:460] Webserver started at http://127.14.106.196:35987/ using document root <none> and password file <none>
I20240501 20:09:24.380784 20629 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:24.380900 20629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:24.381207 20629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:24.383858 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "79e9c50adfeb44929f5d29524169c6c7"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-pg4x"
I20240501 20:09:24.384575 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "79e9c50adfeb44929f5d29524169c6c7"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-pg4x"
I20240501 20:09:24.389617 20629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:09:24.392925 20643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.394220 20629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:24.394379 20629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "79e9c50adfeb44929f5d29524169c6c7"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-pg4x"
I20240501 20:09:24.394565 20629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:24.415961 20629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:24.416762 20629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:24.417025 20629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:24.417917 20629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:24.419425 20629 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:24.419524 20629 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.419641 20629 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:24.419716 20629 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.450464 20629 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:36027
I20240501 20:09:24.450518 20755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:36027 every 8 connection(s)
I20240501 20:09:24.451805 20629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:09:24.460013 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 20629
I20240501 20:09:24.460192 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:09:24.463788 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:0
--local_ip_for_outbound_sockets=127.14.106.197
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:24.464958 20756 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:24.465245 20756 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:24.465975 20756 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:24.467274 20179 ts_manager.cc:194] Registered new tserver with Master: 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196:36027)
I20240501 20:09:24.468030 20179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:55419
I20240501 20:09:24.584327 20496 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
W20240501 20:09:24.783098 20759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:24.783370 20759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:24.783497 20759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:24.790724 20759 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:24.790839 20759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:24.791008 20759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:09:24.799751 20759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:24.800978 20759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:24.802701 20759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:24.814967 20765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:24.815869 20764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:24.816692 20767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:24.817662 20759 server_base.cc:1034] running on GCE node
I20240501 20:09:24.818217 20759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:24.819132 20759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:24.820343 20759 hybrid_clock.cc:648] HybridClock initialized: now 1714594164820291 us; error 45 us; skew 500 ppm
I20240501 20:09:24.820638 20759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:24.822806 20759 webserver.cc:460] Webserver started at http://127.14.106.197:43029/ using document root <none> and password file <none>
I20240501 20:09:24.823474 20759 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:24.823585 20759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:24.823894 20759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:24.826263 20759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a7396c6be769436abac350f7d6e86d55"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-pg4x"
I20240501 20:09:24.827023 20759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a7396c6be769436abac350f7d6e86d55"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-pg4x"
I20240501 20:09:24.831964 20759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20240501 20:09:24.835214 20773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.836473 20759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:09:24.836624 20759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a7396c6be769436abac350f7d6e86d55"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-pg4x"
I20240501 20:09:24.836807 20759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:24.857203 20759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:24.857987 20759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:24.858273 20759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:24.859246 20759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:24.860704 20759 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:24.860807 20759 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.860914 20759 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:24.860996 20759 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.893030 20759 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:35605
I20240501 20:09:24.893101 20885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:35605 every 8 connection(s)
I20240501 20:09:24.894320 20759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:24.901590 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 20759
I20240501 20:09:24.901820 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:09:24.904940 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.198:0
--local_ip_for_outbound_sockets=127.14.106.198
--webserver_interface=127.14.106.198
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:24.907351 20886 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:24.907668 20886 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:24.908532 20886 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:24.909896 20179 ts_manager.cc:194] Registered new tserver with Master: a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
I20240501 20:09:24.910665 20179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:53555
I20240501 20:09:25.026494 20626 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
W20240501 20:09:25.238111 20889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:25.238376 20889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:25.238502 20889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:25.248119 20889 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:25.248323 20889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:25.248519 20889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.198
I20240501 20:09:25.257819 20889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.106.198
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.198
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:25.259166 20889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:25.260850 20889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:25.272632 20894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:25.273449 20895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:25.274284 20897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:25.275138 20889 server_base.cc:1034] running on GCE node
I20240501 20:09:25.275671 20889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:25.276551 20889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:25.277755 20889 hybrid_clock.cc:648] HybridClock initialized: now 1714594165277711 us; error 38 us; skew 500 ppm
I20240501 20:09:25.278059 20889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:25.280222 20889 webserver.cc:460] Webserver started at http://127.14.106.198:43417/ using document root <none> and password file <none>
I20240501 20:09:25.280913 20889 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:25.281028 20889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:25.281327 20889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:25.283902 20889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "133d5e0cbb234be1a5752a784d6d934a"
format_stamp: "Formatted at 2024-05-01 20:09:25 on dist-test-slave-pg4x"
I20240501 20:09:25.284619 20889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "133d5e0cbb234be1a5752a784d6d934a"
format_stamp: "Formatted at 2024-05-01 20:09:25 on dist-test-slave-pg4x"
I20240501 20:09:25.289484 20889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:09:25.292629 20903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:25.293887 20889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:25.294051 20889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "133d5e0cbb234be1a5752a784d6d934a"
format_stamp: "Formatted at 2024-05-01 20:09:25 on dist-test-slave-pg4x"
I20240501 20:09:25.294232 20889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:25.331001 20889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:25.331844 20889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:25.332118 20889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:25.333081 20889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:25.334677 20889 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:25.334797 20889 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:25.334915 20889 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:25.334993 20889 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:25.366535 20889 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.198:43715
I20240501 20:09:25.366607 21015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.198:43715 every 8 connection(s)
I20240501 20:09:25.367965 20889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:09:25.375164 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 20889
I20240501 20:09:25.375331 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:09:25.378536 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.199:0
--local_ip_for_outbound_sockets=127.14.106.199
--webserver_interface=127.14.106.199
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:25.380914 21016 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:25.381222 21016 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:25.381968 21016 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:25.383334 20179 ts_manager.cc:194] Registered new tserver with Master: 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198:43715)
I20240501 20:09:25.384152 20179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.198:49001
I20240501 20:09:25.471019 20756 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
W20240501 20:09:25.714159 21019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:25.714450 21019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:25.714570 21019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:25.721935 21019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:25.722049 21019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:25.722170 21019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.199
I20240501 20:09:25.731266 21019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.106.199
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.199
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:25.732530 21019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:25.734171 21019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:25.747298 21025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:25.748183 21024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:25.749042 21027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:25.749997 21019 server_base.cc:1034] running on GCE node
I20240501 20:09:25.750696 21019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:25.751868 21019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:25.753077 21019 hybrid_clock.cc:648] HybridClock initialized: now 1714594165753029 us; error 39 us; skew 500 ppm
I20240501 20:09:25.753409 21019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:25.755771 21019 webserver.cc:460] Webserver started at http://127.14.106.199:44683/ using document root <none> and password file <none>
I20240501 20:09:25.756498 21019 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:25.756608 21019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:25.756937 21019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:25.759572 21019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "00e1df10b53f4936b747b168af137e4b"
format_stamp: "Formatted at 2024-05-01 20:09:25 on dist-test-slave-pg4x"
I20240501 20:09:25.760303 21019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "00e1df10b53f4936b747b168af137e4b"
format_stamp: "Formatted at 2024-05-01 20:09:25 on dist-test-slave-pg4x"
I20240501 20:09:25.765504 21019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:09:25.768791 21033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:25.770134 21019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:25.770282 21019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "00e1df10b53f4936b747b168af137e4b"
format_stamp: "Formatted at 2024-05-01 20:09:25 on dist-test-slave-pg4x"
I20240501 20:09:25.770463 21019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:25.786942 21019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:25.787715 21019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:25.787992 21019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:25.788885 21019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:25.790313 21019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:25.790413 21019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:25.790526 21019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:25.790597 21019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:25.822801 21019 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.199:39669
I20240501 20:09:25.822854 21145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.199:39669 every 8 connection(s)
I20240501 20:09:25.824126 21019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:25.826854 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 21019
I20240501 20:09:25.827060 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:09:25.836935 21146 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:25.837256 21146 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:25.838027 21146 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:25.839560 20179 ts_manager.cc:194] Registered new tserver with Master: 00e1df10b53f4936b747b168af137e4b (127.14.106.199:39669)
I20240501 20:09:25.840366 20179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.199:58201
I20240501 20:09:25.843796 14763 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:09:25.858809 20179 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46460:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:09:25.888748 20821 tablet_service.cc:1467] Processing CreateTablet for tablet e4dff67082a6473f994c09821a0d7a3d (DEFAULT_TABLE table=TestTable [id=905e41f6a17144258cf5ffc0d8e5e6ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.888748 20300 tablet_service.cc:1467] Processing CreateTablet for tablet e4dff67082a6473f994c09821a0d7a3d (DEFAULT_TABLE table=TestTable [id=905e41f6a17144258cf5ffc0d8e5e6ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.890218 20300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4dff67082a6473f994c09821a0d7a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.890233 20821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4dff67082a6473f994c09821a0d7a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.902812 21162 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:25.904004 21163 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Bootstrap starting.
I20240501 20:09:25.906919 21162 tablet_bootstrap.cc:654] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.907413 21163 tablet_bootstrap.cc:654] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.908242 21162 log.cc:824] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:25.908535 21163 log.cc:824] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:25.910598 21162 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:25.910847 21163 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: No bootstrap required, opened a new log
I20240501 20:09:25.910862 21162 ts_tablet_manager.cc:1367] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240501 20:09:25.911082 21163 ts_tablet_manager.cc:1367] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240501 20:09:25.912993 20886 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
I20240501 20:09:25.917654 21162 raft_consensus.cc:348] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:25.917971 21162 raft_consensus.cc:374] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.918076 21162 raft_consensus.cc:729] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.918701 21162 consensus_queue.cc:260] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:25.918656 21163 raft_consensus.cc:348] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:25.918985 21163 raft_consensus.cc:374] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.919096 21163 raft_consensus.cc:729] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.919720 21163 consensus_queue.cc:260] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:25.920400 21162 ts_tablet_manager.cc:1398] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240501 20:09:25.921221 21163 ts_tablet_manager.cc:1398] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:09:25.995239 21167 raft_consensus.cc:482] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:25.995524 21167 raft_consensus.cc:504] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:25.997138 21167 leader_election.cc:288] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:26.003549 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4dff67082a6473f994c09821a0d7a3d" candidate_uuid: "a7396c6be769436abac350f7d6e86d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:26.003998 20320 raft_consensus.cc:2454] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7396c6be769436abac350f7d6e86d55 in term 0.
I20240501 20:09:26.004618 20774 leader_election.cc:302] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed, a7396c6be769436abac350f7d6e86d55; no voters:
I20240501 20:09:26.004967 21167 raft_consensus.cc:2781] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:26.005122 21167 raft_consensus.cc:482] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:26.005241 21167 raft_consensus.cc:3037] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:26.008059 21167 raft_consensus.cc:504] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:26.008502 21167 leader_election.cc:288] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:26.009008 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4dff67082a6473f994c09821a0d7a3d" candidate_uuid: "a7396c6be769436abac350f7d6e86d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:26.009168 20320 raft_consensus.cc:3037] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:26.011310 20320 raft_consensus.cc:2454] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7396c6be769436abac350f7d6e86d55 in term 1.
I20240501 20:09:26.011706 20774 leader_election.cc:302] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed, a7396c6be769436abac350f7d6e86d55; no voters:
I20240501 20:09:26.011999 21167 raft_consensus.cc:2781] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:26.012429 21167 raft_consensus.cc:686] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 LEADER]: Becoming Leader. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Running, Role: LEADER
I20240501 20:09:26.012837 21167 consensus_queue.cc:237] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:26.016861 20179 catalog_manager.cc:5484] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7396c6be769436abac350f7d6e86d55 (127.14.106.197). New cstate: current_term: 1 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } }
I20240501 20:09:26.031284 14763 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:09:26.033341 14763 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4f8d9458da0b406db99095564f9cf2ed to finish bootstrapping
I20240501 20:09:26.042064 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 28b7e229ab3d4f97a09ca51ac037b008 to finish bootstrapping
I20240501 20:09:26.050138 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2305b26386ac446da2436463b536d296 to finish bootstrapping
I20240501 20:09:26.058177 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 79e9c50adfeb44929f5d29524169c6c7 to finish bootstrapping
I20240501 20:09:26.066110 14763 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a7396c6be769436abac350f7d6e86d55 to finish bootstrapping
I20240501 20:09:26.073400 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 133d5e0cbb234be1a5752a784d6d934a to finish bootstrapping
I20240501 20:09:26.081274 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 00e1df10b53f4936b747b168af137e4b to finish bootstrapping
I20240501 20:09:26.089573 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 20629
I20240501 20:09:26.090479 20751 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:26.120420 20366 tablet.cc:2369] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:26.146734 20887 tablet.cc:2369] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:26.170161 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 20629
I20240501 20:09:26.186049 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 20759
I20240501 20:09:26.187145 20881 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:26.273386 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 20759
I20240501 20:09:26.290130 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 20889
I20240501 20:09:26.291360 21011 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:26.371371 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 20889
I20240501 20:09:26.387632 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 21019
I20240501 20:09:26.388680 21141 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:26.472069 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 21019
I20240501 20:09:27.941704 21181 raft_consensus.cc:482] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:27.941906 21181 raft_consensus.cc:504] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:27.943260 21181 leader_election.cc:288] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
W20240501 20:09:27.944346 20256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.197:35605: connect: Connection refused (error 111)
W20240501 20:09:27.945866 20256 leader_election.cc:334] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Network error: Client connection negotiation failed: client connection to 127.14.106.197:35605: connect: Connection refused (error 111)
I20240501 20:09:27.946040 20256 leader_election.cc:302] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed; no voters: a7396c6be769436abac350f7d6e86d55
I20240501 20:09:27.946380 21181 raft_consensus.cc:2726] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:09:29.833824 21184 raft_consensus.cc:482] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:29.834015 21184 raft_consensus.cc:504] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:29.834599 21184 leader_election.cc:288] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
W20240501 20:09:29.839637 20256 leader_election.cc:334] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Network error: Client connection negotiation failed: client connection to 127.14.106.197:35605: connect: Connection refused (error 111)
I20240501 20:09:29.839813 20256 leader_election.cc:302] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed; no voters: a7396c6be769436abac350f7d6e86d55
I20240501 20:09:29.840056 21184 raft_consensus.cc:2726] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:09:30.239342 20179 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46460:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:30.250365 20300 tablet_service.cc:1467] Processing CreateTablet for tablet afabe33672c647059a9c125c76906fac (DEFAULT_TABLE table=rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.250895 20300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afabe33672c647059a9c125c76906fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.257258 21195 tablet_bootstrap.cc:492] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:30.259218 21195 tablet_bootstrap.cc:654] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.261070 20429 tablet_service.cc:1467] Processing CreateTablet for tablet afabe33672c647059a9c125c76906fac (DEFAULT_TABLE table=rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.262665 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afabe33672c647059a9c125c76906fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.264654 20561 tablet_service.cc:1467] Processing CreateTablet for tablet 62a6957734c44303929840f208f5d1d3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.266073 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a6957734c44303929840f208f5d1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.269791 20430 tablet_service.cc:1467] Processing CreateTablet for tablet 01bf239579d84c07adff90ca8f87824d (DEFAULT_TABLE table=rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.270351 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bf239579d84c07adff90ca8f87824d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.270758 20560 tablet_service.cc:1467] Processing CreateTablet for tablet 01bf239579d84c07adff90ca8f87824d (DEFAULT_TABLE table=rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.260963 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 62a6957734c44303929840f208f5d1d3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.271711 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a6957734c44303929840f208f5d1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.272236 20560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bf239579d84c07adff90ca8f87824d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.282011 21197 tablet_bootstrap.cc:492] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:30.287021 21195 tablet_bootstrap.cc:492] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:30.287230 21195 ts_tablet_manager.cc:1367] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.004s
I20240501 20:09:30.288269 21197 tablet_bootstrap.cc:654] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.288208 21195 raft_consensus.cc:348] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:30.288498 21195 raft_consensus.cc:374] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.288632 21195 raft_consensus.cc:729] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.288911 21195 consensus_queue.cc:260] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:30.289510 21195 ts_tablet_manager.cc:1398] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:30.289831 21198 tablet_bootstrap.cc:492] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:30.290076 21197 log.cc:824] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:30.293078 21197 tablet_bootstrap.cc:492] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:30.293341 21197 ts_tablet_manager.cc:1367] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20240501 20:09:30.294384 21198 tablet_bootstrap.cc:654] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.295361 21198 log.cc:824] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:30.297645 21198 tablet_bootstrap.cc:492] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:30.297881 21198 ts_tablet_manager.cc:1367] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 20:09:30.299630 21197 raft_consensus.cc:348] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.299937 21197 raft_consensus.cc:374] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.300035 21197 raft_consensus.cc:729] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.300664 21197 consensus_queue.cc:260] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.302279 21197 ts_tablet_manager.cc:1398] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20240501 20:09:30.302762 21197 tablet_bootstrap.cc:492] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:30.304386 21197 tablet_bootstrap.cc:654] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.304697 21198 raft_consensus.cc:348] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.305120 21198 raft_consensus.cc:374] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.305235 21198 raft_consensus.cc:729] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.305898 21198 consensus_queue.cc:260] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.308442 21201 raft_consensus.cc:482] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:30.308688 21201 raft_consensus.cc:504] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.309971 21198 ts_tablet_manager.cc:1398] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20240501 20:09:30.310417 21198 tablet_bootstrap.cc:492] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:30.312608 21197 tablet_bootstrap.cc:492] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:30.312748 21197 ts_tablet_manager.cc:1367] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:09:30.312953 21198 tablet_bootstrap.cc:654] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.313529 21197 raft_consensus.cc:348] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.313674 21197 raft_consensus.cc:374] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.313776 21197 raft_consensus.cc:729] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.314121 21197 consensus_queue.cc:260] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.314953 21197 ts_tablet_manager.cc:1398] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:30.316103 21197 tablet_bootstrap.cc:492] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:30.316802 21201 leader_election.cc:288] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:30.317406 21197 tablet_bootstrap.cc:654] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.319463 21197 tablet_bootstrap.cc:492] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:30.319608 21197 ts_tablet_manager.cc:1367] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:30.320417 21197 raft_consensus.cc:348] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:30.320693 21198 tablet_bootstrap.cc:492] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:30.320686 21197 raft_consensus.cc:374] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.320778 21197 raft_consensus.cc:729] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.320837 21198 ts_tablet_manager.cc:1367] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240501 20:09:30.321033 21197 consensus_queue.cc:260] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:30.321498 21197 ts_tablet_manager.cc:1398] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:30.321835 21198 raft_consensus.cc:348] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.322062 21198 raft_consensus.cc:374] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.322162 21198 raft_consensus.cc:729] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.322450 21198 consensus_queue.cc:260] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.322993 21198 ts_tablet_manager.cc:1398] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240501 20:09:30.324753 20497 tablet.cc:2369] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:30.326329 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a6957734c44303929840f208f5d1d3" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296" is_pre_election: true
I20240501 20:09:30.326900 20581 raft_consensus.cc:2454] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:30.327435 20385 leader_election.cc:302] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:30.327739 21201 raft_consensus.cc:2781] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:30.327865 21201 raft_consensus.cc:482] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:30.327977 21201 raft_consensus.cc:3037] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.329993 21201 raft_consensus.cc:504] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.330430 21201 leader_election.cc:288] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:30.330902 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a6957734c44303929840f208f5d1d3" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:09:30.331051 20581 raft_consensus.cc:3037] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.331279 21201 raft_consensus.cc:482] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:30.331425 21201 raft_consensus.cc:504] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.331890 21201 leader_election.cc:288] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:30.332406 20580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bf239579d84c07adff90ca8f87824d" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296" is_pre_election: true
I20240501 20:09:30.332679 20580 raft_consensus.cc:2454] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:30.333004 20385 leader_election.cc:302] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:30.333241 21201 raft_consensus.cc:2781] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:30.333364 21201 raft_consensus.cc:482] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:30.333469 21201 raft_consensus.cc:3037] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.333573 20581 raft_consensus.cc:2454] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:30.333966 20385 leader_election.cc:302] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:30.334250 21206 raft_consensus.cc:2781] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.334877 21206 raft_consensus.cc:686] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:30.335407 21206 consensus_queue.cc:237] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.335851 21201 raft_consensus.cc:504] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.336251 21201 leader_election.cc:288] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:30.337424 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bf239579d84c07adff90ca8f87824d" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:09:30.337636 20581 raft_consensus.cc:3037] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.339715 20179 catalog_manager.cc:5484] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:30.339967 20581 raft_consensus.cc:2454] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:30.340322 20385 leader_election.cc:302] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:30.340564 21206 raft_consensus.cc:2781] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.340708 21206 raft_consensus.cc:686] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:30.340924 21206 consensus_queue.cc:237] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:30.343839 20179 catalog_manager.cc:5484] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:30.412763 21206 raft_consensus.cc:482] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:30.412977 21206 raft_consensus.cc:504] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:30.413606 21206 leader_election.cc:288] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:30.419231 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afabe33672c647059a9c125c76906fac" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:30.419486 20320 raft_consensus.cc:2454] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:30.419970 20384 leader_election.cc:302] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:30.420249 21206 raft_consensus.cc:2781] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:30.420374 21206 raft_consensus.cc:482] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:30.420442 21206 raft_consensus.cc:3037] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.422286 21206 raft_consensus.cc:504] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:30.422868 21206 leader_election.cc:288] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:30.423388 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afabe33672c647059a9c125c76906fac" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:30.423591 20320 raft_consensus.cc:3037] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.425416 20320 raft_consensus.cc:2454] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:30.425781 20384 leader_election.cc:302] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:30.426064 21206 raft_consensus.cc:2781] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.426191 21206 raft_consensus.cc:686] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:30.426445 21206 consensus_queue.cc:237] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:30.428967 20179 catalog_manager.cc:5484] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } }
W20240501 20:09:30.512547 20627 tablet.cc:2369] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:30.746956 21206 consensus_queue.cc:1035] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:30.773975 21213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:30.774284 21213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:30.781589 21213 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 20:09:30.817909 21201 consensus_queue.cc:1035] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:30.885593 21201 consensus_queue.cc:1035] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 193.26 ms
time per row: 0.09663 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:32.046073 20177 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46460:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:32.054811 20299 tablet_service.cc:1467] Processing CreateTablet for tablet b9ccf34e292a480aa7283631edb5a0d2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.054731 20300 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff0eebf61824be1aff2cf5c43f45d78 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.055295 20300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ff0eebf61824be1aff2cf5c43f45d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.055640 20560 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff0eebf61824be1aff2cf5c43f45d78 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.055598 20561 tablet_service.cc:1467] Processing CreateTablet for tablet b9ccf34e292a480aa7283631edb5a0d2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.056054 20560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ff0eebf61824be1aff2cf5c43f45d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.055965 20431 tablet_service.cc:1467] Processing CreateTablet for tablet f9480642e71d46679005d4f55d2e88f8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.056358 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9480642e71d46679005d4f55d2e88f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.059612 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ccf34e292a480aa7283631edb5a0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.059499 20298 tablet_service.cc:1467] Processing CreateTablet for tablet f9480642e71d46679005d4f55d2e88f8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.059942 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9480642e71d46679005d4f55d2e88f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.063122 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ccf34e292a480aa7283631edb5a0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.071460 21263 tablet_bootstrap.cc:492] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:32.073462 21263 tablet_bootstrap.cc:654] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.074895 21265 tablet_bootstrap.cc:492] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:32.075102 21264 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:32.076526 21265 tablet_bootstrap.cc:654] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.076649 21264 tablet_bootstrap.cc:654] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.077730 21263 tablet_bootstrap.cc:492] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:32.077926 21263 ts_tablet_manager.cc:1367] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:09:32.078671 21265 tablet_bootstrap.cc:492] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:32.078666 21264 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:32.078809 21265 ts_tablet_manager.cc:1367] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:32.078810 21264 ts_tablet_manager.cc:1367] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 20:09:32.078692 21263 raft_consensus.cc:348] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.078896 21263 raft_consensus.cc:374] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.079008 21263 raft_consensus.cc:729] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.079234 21263 consensus_queue.cc:260] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.079447 21265 raft_consensus.cc:348] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.079680 21265 raft_consensus.cc:374] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.079802 21265 raft_consensus.cc:729] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.079862 21264 raft_consensus.cc:348] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:32.080149 21264 raft_consensus.cc:374] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.080075 21265 consensus_queue.cc:260] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.080404 21264 raft_consensus.cc:729] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.080677 21264 consensus_queue.cc:260] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:32.080862 21263 ts_tablet_manager.cc:1398] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:32.081133 21263 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:32.081214 21264 ts_tablet_manager.cc:1398] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:32.082489 21263 tablet_bootstrap.cc:654] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.085029 21263 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:32.085170 21263 ts_tablet_manager.cc:1367] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:32.085781 21263 raft_consensus.cc:348] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:32.085940 21263 raft_consensus.cc:374] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.086021 21263 raft_consensus.cc:729] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.086239 21263 consensus_queue.cc:260] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:32.086704 21263 ts_tablet_manager.cc:1398] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:09:32.087013 21263 tablet_bootstrap.cc:492] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:32.087055 21265 ts_tablet_manager.cc:1398] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20240501 20:09:32.087356 21265 tablet_bootstrap.cc:492] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:32.088516 21263 tablet_bootstrap.cc:654] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.090207 21265 tablet_bootstrap.cc:654] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.090590 21263 tablet_bootstrap.cc:492] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:32.090749 21263 ts_tablet_manager.cc:1367] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:32.091447 21263 raft_consensus.cc:348] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.091635 21263 raft_consensus.cc:374] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.091738 21263 raft_consensus.cc:729] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.092015 21265 tablet_bootstrap.cc:492] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:32.091977 21263 consensus_queue.cc:260] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.092164 21265 ts_tablet_manager.cc:1367] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:32.092518 21263 ts_tablet_manager.cc:1398] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:32.092847 21265 raft_consensus.cc:348] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.093010 21265 raft_consensus.cc:374] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.093096 21265 raft_consensus.cc:729] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.093308 21265 consensus_queue.cc:260] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.093724 21265 ts_tablet_manager.cc:1398] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:32.111106 21206 raft_consensus.cc:482] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.111258 21206 raft_consensus.cc:504] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:32.111730 21206 leader_election.cc:288] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:32.112180 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9480642e71d46679005d4f55d2e88f8" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:32.112376 20320 raft_consensus.cc:2454] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:32.112756 20384 leader_election.cc:302] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:32.113004 21206 raft_consensus.cc:2781] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.113113 21206 raft_consensus.cc:482] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.113204 21206 raft_consensus.cc:3037] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.115257 21206 raft_consensus.cc:504] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:32.115667 21206 leader_election.cc:288] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:32.116210 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9480642e71d46679005d4f55d2e88f8" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:32.116394 20320 raft_consensus.cc:3037] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.119225 20320 raft_consensus.cc:2454] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:32.119570 20384 leader_election.cc:302] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:32.119844 21206 raft_consensus.cc:2781] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.120232 21206 raft_consensus.cc:686] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:32.120517 21206 consensus_queue.cc:237] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:32.122689 20179 catalog_manager.cc:5484] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } }
I20240501 20:09:32.270442 21269 raft_consensus.cc:482] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.270612 21269 raft_consensus.cc:504] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.271585 21269 leader_election.cc:288] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:32.277295 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccf34e292a480aa7283631edb5a0d2" candidate_uuid: "4f8d9458da0b406db99095564f9cf2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296" is_pre_election: true
I20240501 20:09:32.277519 20581 raft_consensus.cc:2454] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9458da0b406db99095564f9cf2ed in term 0.
I20240501 20:09:32.277972 20254 leader_election.cc:302] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:32.278261 21269 raft_consensus.cc:2781] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.278373 21269 raft_consensus.cc:482] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.278463 21269 raft_consensus.cc:3037] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.280403 21269 raft_consensus.cc:504] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.280894 21269 leader_election.cc:288] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 1 election: Requested vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:32.281353 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccf34e292a480aa7283631edb5a0d2" candidate_uuid: "4f8d9458da0b406db99095564f9cf2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:09:32.281525 20581 raft_consensus.cc:3037] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.283382 20581 raft_consensus.cc:2454] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9458da0b406db99095564f9cf2ed in term 1.
I20240501 20:09:32.283746 20254 leader_election.cc:302] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:32.284041 21269 raft_consensus.cc:2781] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.284437 21269 raft_consensus.cc:686] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Becoming Leader. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Running, Role: LEADER
I20240501 20:09:32.284840 21269 consensus_queue.cc:237] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.287603 20179 catalog_manager.cc:5484] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4f8d9458da0b406db99095564f9cf2ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:32.444994 21269 raft_consensus.cc:482] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.445202 21269 raft_consensus.cc:504] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.445680 21269 leader_election.cc:288] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:32.446216 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ff0eebf61824be1aff2cf5c43f45d78" candidate_uuid: "4f8d9458da0b406db99095564f9cf2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296" is_pre_election: true
I20240501 20:09:32.446449 20581 raft_consensus.cc:2454] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9458da0b406db99095564f9cf2ed in term 0.
I20240501 20:09:32.446835 20254 leader_election.cc:302] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:32.447135 21269 raft_consensus.cc:2781] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.447225 21269 raft_consensus.cc:482] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.447331 21269 raft_consensus.cc:3037] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.449277 21269 raft_consensus.cc:504] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.449693 21269 leader_election.cc:288] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 1 election: Requested vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:32.450145 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ff0eebf61824be1aff2cf5c43f45d78" candidate_uuid: "4f8d9458da0b406db99095564f9cf2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:09:32.450338 20581 raft_consensus.cc:3037] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.452194 20581 raft_consensus.cc:2454] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9458da0b406db99095564f9cf2ed in term 1.
I20240501 20:09:32.452593 20254 leader_election.cc:302] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:32.452863 21269 raft_consensus.cc:2781] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.453028 21269 raft_consensus.cc:686] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Becoming Leader. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Running, Role: LEADER
I20240501 20:09:32.453413 21269 consensus_queue.cc:237] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:32.455829 20179 catalog_manager.cc:5484] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4f8d9458da0b406db99095564f9cf2ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:32.712904 21269 consensus_queue.cc:1035] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:32.728474 21206 consensus_queue.cc:1035] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:32.845615 21281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:32.845950 21281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:32.853145 21281 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:09:32.891424 21278 consensus_queue.cc:1035] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:33.030933 21269 raft_consensus.cc:482] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:33.031142 21269 raft_consensus.cc:504] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:33.031668 21269 leader_election.cc:288] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
INSERT report
rows total: 2000
time total: 155.804 ms
time per row: 0.0779019 ms
W20240501 20:09:33.059732 20256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.197:35605: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:09:33.070298 20256 leader_election.cc:334] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Network error: Client connection negotiation failed: client connection to 127.14.106.197:35605: connect: Connection refused (error 111)
I20240501 20:09:33.070503 20256 leader_election.cc:302] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed; no voters: a7396c6be769436abac350f7d6e86d55
I20240501 20:09:33.070827 21269 raft_consensus.cc:2726] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:09:33.152411 20179 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46460:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:33.160732 20431 tablet_service.cc:1467] Processing CreateTablet for tablet e8477d3f66b940538bb74f2cf87a3d47 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:33.160722 20560 tablet_service.cc:1467] Processing CreateTablet for tablet e8477d3f66b940538bb74f2cf87a3d47 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:33.161162 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8477d3f66b940538bb74f2cf87a3d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:33.161152 20560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8477d3f66b940538bb74f2cf87a3d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:33.161815 20299 tablet_service.cc:1467] Processing CreateTablet for tablet 4c862c7f6ec24fe2916c97cc7c72401b (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:33.162238 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c862c7f6ec24fe2916c97cc7c72401b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:33.162133 20561 tablet_service.cc:1467] Processing CreateTablet for tablet e4c119cb173942819ec01f7f3797c9d7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:33.162544 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c119cb173942819ec01f7f3797c9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:33.166292 21336 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:33.166245 20429 tablet_service.cc:1467] Processing CreateTablet for tablet 4c862c7f6ec24fe2916c97cc7c72401b (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:33.166615 20430 tablet_service.cc:1467] Processing CreateTablet for tablet e4c119cb173942819ec01f7f3797c9d7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:33.168270 21336 tablet_bootstrap.cc:654] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:33.167564 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c862c7f6ec24fe2916c97cc7c72401b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:33.171275 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c119cb173942819ec01f7f3797c9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:33.182230 21339 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:33.185153 21339 tablet_bootstrap.cc:654] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:33.199156 21337 tablet_bootstrap.cc:492] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:33.200078 21339 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:33.200292 21339 ts_tablet_manager.cc:1367] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.013s
I20240501 20:09:33.200333 21336 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:33.200587 21336 ts_tablet_manager.cc:1367] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.020s
I20240501 20:09:33.201088 21337 tablet_bootstrap.cc:654] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:33.201282 21339 raft_consensus.cc:348] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.201501 21339 raft_consensus.cc:374] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:33.201467 21336 raft_consensus.cc:348] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.201623 21339 raft_consensus.cc:729] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:33.201678 21336 raft_consensus.cc:374] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:33.201805 21336 raft_consensus.cc:729] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:33.201874 21339 consensus_queue.cc:260] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.202062 21336 consensus_queue.cc:260] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.202790 21336 ts_tablet_manager.cc:1398] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:33.203083 21336 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:33.203193 21339 ts_tablet_manager.cc:1398] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 20:09:33.203505 21339 tablet_bootstrap.cc:492] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:33.203846 21337 tablet_bootstrap.cc:492] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:33.204072 21337 ts_tablet_manager.cc:1367] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:09:33.205525 21339 tablet_bootstrap.cc:654] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:33.206394 21336 tablet_bootstrap.cc:654] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:33.205068 21337 raft_consensus.cc:348] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:33.206737 21337 raft_consensus.cc:374] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:33.206874 21337 raft_consensus.cc:729] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:33.207111 21337 consensus_queue.cc:260] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:33.207782 21337 ts_tablet_manager.cc:1398] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:33.209554 21339 tablet_bootstrap.cc:492] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:33.209736 21339 ts_tablet_manager.cc:1367] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240501 20:09:33.210429 21336 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:33.210597 21336 ts_tablet_manager.cc:1367] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240501 20:09:33.210520 21339 raft_consensus.cc:348] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:33.210741 21339 raft_consensus.cc:374] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:33.210820 21339 raft_consensus.cc:729] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:33.211032 21339 consensus_queue.cc:260] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:33.211402 21336 raft_consensus.cc:348] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.211540 21339 ts_tablet_manager.cc:1398] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:33.211601 21336 raft_consensus.cc:374] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:33.211665 21336 raft_consensus.cc:729] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:33.211812 21339 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:33.211891 21336 consensus_queue.cc:260] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.212380 21336 ts_tablet_manager.cc:1398] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:33.213255 21339 tablet_bootstrap.cc:654] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:33.215346 21339 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:33.215489 21339 ts_tablet_manager.cc:1367] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:33.216104 21339 raft_consensus.cc:348] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.216253 21339 raft_consensus.cc:374] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:33.216336 21339 raft_consensus.cc:729] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:33.216521 21339 consensus_queue.cc:260] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.216909 21339 ts_tablet_manager.cc:1398] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:33.255466 21282 raft_consensus.cc:482] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:33.255672 21282 raft_consensus.cc:504] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.257025 21282 leader_election.cc:288] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:33.262902 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8477d3f66b940538bb74f2cf87a3d47" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008" is_pre_election: true
I20240501 20:09:33.263286 20451 raft_consensus.cc:2454] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 0.
I20240501 20:09:33.263788 20515 leader_election.cc:302] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:33.264156 21282 raft_consensus.cc:2781] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:33.264261 21282 raft_consensus.cc:482] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:33.264356 21282 raft_consensus.cc:3037] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:33.266322 21282 raft_consensus.cc:504] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.266808 21282 leader_election.cc:288] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 election: Requested vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:33.267258 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8477d3f66b940538bb74f2cf87a3d47" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
I20240501 20:09:33.267432 20451 raft_consensus.cc:3037] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:33.269461 20451 raft_consensus.cc:2454] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 1.
I20240501 20:09:33.269796 20515 leader_election.cc:302] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:33.270092 21282 raft_consensus.cc:2781] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:33.270502 21282 raft_consensus.cc:686] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Becoming Leader. State: Replica: 2305b26386ac446da2436463b536d296, State: Running, Role: LEADER
I20240501 20:09:33.271162 21287 raft_consensus.cc:482] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:33.271121 21282 consensus_queue.cc:237] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.271277 21287 raft_consensus.cc:504] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:33.271760 21287 leader_election.cc:288] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:33.272275 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c862c7f6ec24fe2916c97cc7c72401b" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:33.272504 20320 raft_consensus.cc:2454] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:33.272914 20384 leader_election.cc:302] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:33.273303 21287 raft_consensus.cc:2781] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:33.273416 21287 raft_consensus.cc:482] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:33.273522 21287 raft_consensus.cc:3037] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:33.275393 21287 raft_consensus.cc:504] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:33.275542 20179 catalog_manager.cc:5484] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2305b26386ac446da2436463b536d296 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:33.275789 21287 leader_election.cc:288] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:33.276208 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c862c7f6ec24fe2916c97cc7c72401b" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:33.276407 20320 raft_consensus.cc:3037] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:33.278685 20320 raft_consensus.cc:2454] T 4c862c7f6ec24fe2916c97cc7c72401b P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:33.279037 20384 leader_election.cc:302] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:33.279294 21287 raft_consensus.cc:2781] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:33.279440 21287 raft_consensus.cc:686] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:33.279670 21287 consensus_queue.cc:237] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:33.281903 20179 catalog_manager.cc:5484] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } }
I20240501 20:09:33.465263 21287 raft_consensus.cc:482] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:33.465495 21287 raft_consensus.cc:504] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.466040 21287 leader_election.cc:288] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:33.466651 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c119cb173942819ec01f7f3797c9d7" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296" is_pre_election: true
I20240501 20:09:33.466902 20581 raft_consensus.cc:2454] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:33.467365 20385 leader_election.cc:302] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:33.467649 21287 raft_consensus.cc:2781] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:33.467749 21287 raft_consensus.cc:482] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:33.467834 21287 raft_consensus.cc:3037] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:33.469653 21287 raft_consensus.cc:504] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.470165 21287 leader_election.cc:288] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:33.470681 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c119cb173942819ec01f7f3797c9d7" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:09:33.470866 20581 raft_consensus.cc:3037] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:33.472738 20581 raft_consensus.cc:2454] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:33.473110 20385 leader_election.cc:302] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:33.473397 21287 raft_consensus.cc:2781] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:33.473580 21287 raft_consensus.cc:686] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:33.473838 21287 consensus_queue.cc:237] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:33.476140 20179 catalog_manager.cc:5484] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:33.852102 21287 consensus_queue.cc:1035] T 4c862c7f6ec24fe2916c97cc7c72401b P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:33.859692 21352 consensus_queue.cc:1035] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:33.872514 21350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:33.872808 21350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:33.880252 21350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 20:09:33.922096 21287 consensus_queue.cc:1035] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 189.014 ms
time per row: 0.094507 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:35.138887 20177 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46460:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:35.147203 20299 tablet_service.cc:1467] Processing CreateTablet for tablet 86d6707287cc48169ae7a548c1c01597 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.147365 20561 tablet_service.cc:1467] Processing CreateTablet for tablet 86d6707287cc48169ae7a548c1c01597 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.147660 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d6707287cc48169ae7a548c1c01597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.147764 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d6707287cc48169ae7a548c1c01597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.148284 20298 tablet_service.cc:1467] Processing CreateTablet for tablet 5a45309550b449c1bf0ba233c24140ed (DEFAULT_TABLE table=rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.148419 20430 tablet_service.cc:1467] Processing CreateTablet for tablet ee446411ba4e4da2b295080da28d40b9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.148681 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a45309550b449c1bf0ba233c24140ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.148821 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee446411ba4e4da2b295080da28d40b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.151087 20300 tablet_service.cc:1467] Processing CreateTablet for tablet ee446411ba4e4da2b295080da28d40b9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.151466 20300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee446411ba4e4da2b295080da28d40b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.152005 20560 tablet_service.cc:1467] Processing CreateTablet for tablet 5a45309550b449c1bf0ba233c24140ed (DEFAULT_TABLE table=rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.152359 20560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a45309550b449c1bf0ba233c24140ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.162695 21409 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:35.162977 21411 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:35.164106 21412 tablet_bootstrap.cc:492] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:35.164217 21409 tablet_bootstrap.cc:654] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.164750 21411 tablet_bootstrap.cc:654] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.167163 21412 tablet_bootstrap.cc:654] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.167181 21409 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:35.167311 21409 ts_tablet_manager.cc:1367] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:09:35.167366 21411 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:35.167513 21411 ts_tablet_manager.cc:1367] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 20:09:35.167899 21409 raft_consensus.cc:348] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.168043 21409 raft_consensus.cc:374] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.168138 21409 raft_consensus.cc:729] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.168290 21411 raft_consensus.cc:348] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.168473 21411 raft_consensus.cc:374] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.168378 21409 consensus_queue.cc:260] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.168567 21411 raft_consensus.cc:729] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.168790 21411 consensus_queue.cc:260] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.168907 21409 ts_tablet_manager.cc:1398] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:35.169276 21412 tablet_bootstrap.cc:492] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:35.169325 21409 tablet_bootstrap.cc:492] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:35.169481 21412 ts_tablet_manager.cc:1367] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:09:35.170284 21411 ts_tablet_manager.cc:1398] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:35.170280 21412 raft_consensus.cc:348] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:35.170557 21412 raft_consensus.cc:374] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.170598 21411 tablet_bootstrap.cc:492] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:35.170746 21412 raft_consensus.cc:729] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.171020 21412 consensus_queue.cc:260] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:35.171723 21412 ts_tablet_manager.cc:1398] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:35.172039 21409 tablet_bootstrap.cc:654] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.172964 21411 tablet_bootstrap.cc:654] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.174973 21411 tablet_bootstrap.cc:492] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:35.175144 21409 tablet_bootstrap.cc:492] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:35.175199 21411 ts_tablet_manager.cc:1367] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:35.175348 21409 ts_tablet_manager.cc:1367] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:09:35.175992 21411 raft_consensus.cc:348] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.176251 21411 raft_consensus.cc:374] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.176173 21409 raft_consensus.cc:348] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.176330 21411 raft_consensus.cc:729] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.176367 21409 raft_consensus.cc:374] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.176463 21409 raft_consensus.cc:729] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.176599 21411 consensus_queue.cc:260] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.176688 21409 consensus_queue.cc:260] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.177181 21409 ts_tablet_manager.cc:1398] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:35.177431 21409 tablet_bootstrap.cc:492] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:35.177781 21411 ts_tablet_manager.cc:1398] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:35.179101 21409 tablet_bootstrap.cc:654] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.180846 21409 tablet_bootstrap.cc:492] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:35.180979 21409 ts_tablet_manager.cc:1367] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:35.181499 21409 raft_consensus.cc:348] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:35.181628 21409 raft_consensus.cc:374] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.181733 21409 raft_consensus.cc:729] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.181957 21409 consensus_queue.cc:260] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:35.182343 21409 ts_tablet_manager.cc:1398] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:35.407452 21357 raft_consensus.cc:482] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:35.407652 21357 raft_consensus.cc:504] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.408304 21357 leader_election.cc:288] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:35.413913 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d6707287cc48169ae7a548c1c01597" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:35.414127 20320 raft_consensus.cc:2454] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 0.
I20240501 20:09:35.414525 20514 leader_election.cc:302] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:35.414861 21357 raft_consensus.cc:2781] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:35.415004 21357 raft_consensus.cc:482] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:35.415115 21357 raft_consensus.cc:3037] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.417008 21357 raft_consensus.cc:504] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.417507 21357 leader_election.cc:288] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:35.418102 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d6707287cc48169ae7a548c1c01597" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:35.418279 20320 raft_consensus.cc:3037] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.420310 20320 raft_consensus.cc:2454] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 1.
I20240501 20:09:35.420718 20514 leader_election.cc:302] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:35.421025 21357 raft_consensus.cc:2781] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:35.421375 21357 raft_consensus.cc:686] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Becoming Leader. State: Replica: 2305b26386ac446da2436463b536d296, State: Running, Role: LEADER
I20240501 20:09:35.421627 21357 consensus_queue.cc:237] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.423985 20173 catalog_manager.cc:5484] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2305b26386ac446da2436463b536d296 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:35.479210 21287 raft_consensus.cc:482] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:35.479415 21287 raft_consensus.cc:504] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:35.479914 21287 leader_election.cc:288] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:35.480468 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee446411ba4e4da2b295080da28d40b9" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:35.480695 20320 raft_consensus.cc:2454] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:35.481194 20384 leader_election.cc:302] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:35.481501 21287 raft_consensus.cc:2781] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:35.481616 21287 raft_consensus.cc:482] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:35.481705 21287 raft_consensus.cc:3037] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.483542 21287 raft_consensus.cc:504] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:35.483963 21287 leader_election.cc:288] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:35.484407 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee446411ba4e4da2b295080da28d40b9" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:35.484592 20320 raft_consensus.cc:3037] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.486385 20320 raft_consensus.cc:2454] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:35.486766 20384 leader_election.cc:302] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:35.487062 21287 raft_consensus.cc:2781] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:35.487198 21287 raft_consensus.cc:686] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:35.487463 21287 consensus_queue.cc:237] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:35.489679 20173 catalog_manager.cc:5484] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } }
I20240501 20:09:35.505597 21357 raft_consensus.cc:482] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:35.505797 21357 raft_consensus.cc:504] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.506258 21357 leader_election.cc:288] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:35.506867 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:35.507081 20320 raft_consensus.cc:2454] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 0.
I20240501 20:09:35.507476 20514 leader_election.cc:302] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:35.507731 21357 raft_consensus.cc:2781] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:35.507812 21357 raft_consensus.cc:482] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:35.507896 21357 raft_consensus.cc:3037] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.509685 21357 raft_consensus.cc:504] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.510097 21357 leader_election.cc:288] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:35.510615 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:35.510795 20320 raft_consensus.cc:3037] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.512615 20320 raft_consensus.cc:2454] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 1.
I20240501 20:09:35.512990 20514 leader_election.cc:302] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:35.513265 21357 raft_consensus.cc:2781] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:35.513427 21357 raft_consensus.cc:686] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Becoming Leader. State: Replica: 2305b26386ac446da2436463b536d296, State: Running, Role: LEADER
I20240501 20:09:35.513657 21357 consensus_queue.cc:237] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:35.516048 20173 catalog_manager.cc:5484] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2305b26386ac446da2436463b536d296 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:35.820585 21357 consensus_queue.cc:1035] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:35.852905 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:35.853199 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:35.860548 21425 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 20:09:35.981268 21287 consensus_queue.cc:1035] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:35.989161 20319 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:35.994544 21357 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.074761 21463 mvcc.cc:205] Tried to move back new op lower bound from 7022977744805851136 to 7022977742905393152. Current Snapshot: MvccSnapshot[applied={T|T < 7022977744805851136}]
INSERT report
rows total: 2000
time total: 187.417 ms
time per row: 0.0937087 ms
I20240501 20:09:36.116182 21269 raft_consensus.cc:482] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.116402 21269 raft_consensus.cc:504] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:36.116987 21269 leader_election.cc:288] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
W20240501 20:09:36.119570 20256 leader_election.cc:334] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Network error: Client connection negotiation failed: client connection to 127.14.106.197:35605: connect: Connection refused (error 111)
I20240501 20:09:36.119732 20256 leader_election.cc:302] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed; no voters: a7396c6be769436abac350f7d6e86d55
I20240501 20:09:36.120043 21269 raft_consensus.cc:2726] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:09:36.192591 20172 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46460:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:36.202019 20560 tablet_service.cc:1467] Processing CreateTablet for tablet e74fb1e0e017408f946a8df8ae304f13 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.202019 20430 tablet_service.cc:1467] Processing CreateTablet for tablet e74fb1e0e017408f946a8df8ae304f13 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.202472 20560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74fb1e0e017408f946a8df8ae304f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.202472 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74fb1e0e017408f946a8df8ae304f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.202860 20298 tablet_service.cc:1467] Processing CreateTablet for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.203162 20561 tablet_service.cc:1467] Processing CreateTablet for tablet 2e414a3678e646c6b5c6c0dad4fac30e (DEFAULT_TABLE table=rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.203269 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.203567 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e414a3678e646c6b5c6c0dad4fac30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.207506 20429 tablet_service.cc:1467] Processing CreateTablet for tablet 2e414a3678e646c6b5c6c0dad4fac30e (DEFAULT_TABLE table=rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.207540 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.207890 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e414a3678e646c6b5c6c0dad4fac30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.209921 21483 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: Bootstrap starting.
I20240501 20:09:36.211126 21484 tablet_bootstrap.cc:492] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:36.211156 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.211737 21483 tablet_bootstrap.cc:654] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.212682 21485 tablet_bootstrap.cc:492] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:36.212946 21484 tablet_bootstrap.cc:654] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.216358 21485 tablet_bootstrap.cc:654] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.221161 21484 tablet_bootstrap.cc:492] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:36.221416 21484 ts_tablet_manager.cc:1367] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240501 20:09:36.222323 21484 raft_consensus.cc:348] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:36.222615 21484 raft_consensus.cc:374] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.222786 21484 raft_consensus.cc:729] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.223181 21484 consensus_queue.cc:260] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:36.224133 21484 ts_tablet_manager.cc:1398] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:36.226897 21484 tablet_bootstrap.cc:492] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296: Bootstrap starting.
I20240501 20:09:36.229041 21484 tablet_bootstrap.cc:654] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.232084 21484 tablet_bootstrap.cc:492] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296: No bootstrap required, opened a new log
I20240501 20:09:36.232190 21485 tablet_bootstrap.cc:492] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:36.232249 21484 ts_tablet_manager.cc:1367] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 20:09:36.232307 21483 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: No bootstrap required, opened a new log
I20240501 20:09:36.232337 21485 ts_tablet_manager.cc:1367] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.007s
I20240501 20:09:36.232506 21483 ts_tablet_manager.cc:1367] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.019s
I20240501 20:09:36.233060 21484 raft_consensus.cc:348] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.233110 21483 raft_consensus.cc:348] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.233078 21485 raft_consensus.cc:348] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.233255 21483 raft_consensus.cc:374] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.233259 21485 raft_consensus.cc:374] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.233274 21484 raft_consensus.cc:374] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.233342 21484 raft_consensus.cc:729] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.233355 21485 raft_consensus.cc:729] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.233376 21483 raft_consensus.cc:729] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.233599 21483 consensus_queue.cc:260] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.233587 21485 consensus_queue.cc:260] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.233610 21484 consensus_queue.cc:260] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.234201 21484 ts_tablet_manager.cc:1398] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:36.234263 21485 ts_tablet_manager.cc:1398] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:36.234530 21485 tablet_bootstrap.cc:492] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:36.234959 21483 ts_tablet_manager.cc:1398] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:36.237463 21485 tablet_bootstrap.cc:654] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.239408 21485 tablet_bootstrap.cc:492] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:36.239547 21485 ts_tablet_manager.cc:1367] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:09:36.240080 21485 raft_consensus.cc:348] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:36.240240 21485 raft_consensus.cc:374] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.240314 21485 raft_consensus.cc:729] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.240520 21485 consensus_queue.cc:260] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:36.240962 21485 ts_tablet_manager.cc:1398] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:36.241204 21485 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: Bootstrap starting.
I20240501 20:09:36.242265 21485 tablet_bootstrap.cc:654] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.243849 21485 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: No bootstrap required, opened a new log
I20240501 20:09:36.243997 21485 ts_tablet_manager.cc:1367] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:36.244661 21485 raft_consensus.cc:348] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.244824 21485 raft_consensus.cc:374] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.244944 21485 raft_consensus.cc:729] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.245160 21485 consensus_queue.cc:260] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.245515 21485 ts_tablet_manager.cc:1398] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:36.283600 21287 raft_consensus.cc:482] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.283778 21287 raft_consensus.cc:504] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:36.284232 21287 leader_election.cc:288] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:36.284765 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e414a3678e646c6b5c6c0dad4fac30e" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296" is_pre_election: true
I20240501 20:09:36.284996 20581 raft_consensus.cc:2454] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:36.285432 20385 leader_election.cc:302] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:36.285725 21287 raft_consensus.cc:2781] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.285840 21287 raft_consensus.cc:482] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.285926 21287 raft_consensus.cc:3037] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.287932 21287 raft_consensus.cc:504] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:36.288378 21287 leader_election.cc:288] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:36.288964 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e414a3678e646c6b5c6c0dad4fac30e" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:09:36.289170 20581 raft_consensus.cc:3037] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.291064 20581 raft_consensus.cc:2454] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:36.291455 20385 leader_election.cc:302] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:36.291765 21287 raft_consensus.cc:2781] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.291937 21287 raft_consensus.cc:686] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:36.292243 21287 consensus_queue.cc:237] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } }
I20240501 20:09:36.294555 20174 catalog_manager.cc:5484] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } }
I20240501 20:09:36.426889 21287 raft_consensus.cc:482] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.427129 21287 raft_consensus.cc:504] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.427646 21287 leader_election.cc:288] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:36.428215 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:36.428409 20319 raft_consensus.cc:2454] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 0.
I20240501 20:09:36.428843 20384 leader_election.cc:302] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:36.429140 21287 raft_consensus.cc:2781] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.429244 21287 raft_consensus.cc:482] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.429356 21287 raft_consensus.cc:3037] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.431423 21287 raft_consensus.cc:504] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.431849 21287 leader_election.cc:288] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:36.432432 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4" candidate_uuid: "28b7e229ab3d4f97a09ca51ac037b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:36.432613 20319 raft_consensus.cc:3037] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.434453 20319 raft_consensus.cc:2454] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7e229ab3d4f97a09ca51ac037b008 in term 1.
I20240501 20:09:36.434893 20384 leader_election.cc:302] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:09:36.435130 21287 raft_consensus.cc:2781] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.435302 21287 raft_consensus.cc:686] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:36.435591 21287 consensus_queue.cc:237] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.438838 20174 catalog_manager.cc:5484] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } }
I20240501 20:09:36.446067 21357 raft_consensus.cc:482] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.446259 21357 raft_consensus.cc:504] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.446736 21357 leader_election.cc:288] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:36.447228 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74fb1e0e017408f946a8df8ae304f13" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008" is_pre_election: true
I20240501 20:09:36.447440 20450 raft_consensus.cc:2454] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 0.
I20240501 20:09:36.447894 20515 leader_election.cc:302] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:36.448257 21357 raft_consensus.cc:2781] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.448356 21357 raft_consensus.cc:482] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.448473 21357 raft_consensus.cc:3037] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.450392 21357 raft_consensus.cc:504] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.450824 21357 leader_election.cc:288] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 election: Requested vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:36.451243 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74fb1e0e017408f946a8df8ae304f13" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
I20240501 20:09:36.451400 20450 raft_consensus.cc:3037] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.453155 20450 raft_consensus.cc:2454] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 1.
I20240501 20:09:36.453490 20515 leader_election.cc:302] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:36.453758 21357 raft_consensus.cc:2781] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.453912 21357 raft_consensus.cc:686] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Becoming Leader. State: Replica: 2305b26386ac446da2436463b536d296, State: Running, Role: LEADER
I20240501 20:09:36.454149 21357 consensus_queue.cc:237] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } }
I20240501 20:09:36.456665 20174 catalog_manager.cc:5484] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2305b26386ac446da2436463b536d296 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } }
W20240501 20:09:36.834487 21495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:36.834795 21495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:36.842311 21495 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 20:09:36.908286 21287 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.910274 21466 consensus_queue.cc:1035] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.963045 21357 consensus_queue.cc:1035] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 219.589 ms
time per row: 0.109794 ms
I20240501 20:09:37.199044 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:36027
--local_ip_for_outbound_sockets=127.14.106.196
--tserver_master_addrs=127.14.106.254:36189
--webserver_port=35987
--webserver_interface=127.14.106.196
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:09:37.524235 21546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:37.524530 21546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:37.524632 21546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:37.532104 21546 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:37.532220 21546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:37.532335 21546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:09:37.541528 21546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:36027
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=35987
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:37.542827 21546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:37.544621 21546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:37.557421 21551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:37.558218 21552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:37.559026 21554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:37.559924 21546 server_base.cc:1034] running on GCE node
I20240501 20:09:37.560497 21546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:37.561403 21546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:37.562671 21546 hybrid_clock.cc:648] HybridClock initialized: now 1714594177562604 us; error 34 us; skew 500 ppm
I20240501 20:09:37.563045 21546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:37.565721 21546 webserver.cc:460] Webserver started at http://127.14.106.196:35987/ using document root <none> and password file <none>
I20240501 20:09:37.566576 21546 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:37.566777 21546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:37.576587 21546 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.006s
I20240501 20:09:37.579849 21560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:37.581167 21546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:37.581316 21546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "79e9c50adfeb44929f5d29524169c6c7"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-pg4x"
I20240501 20:09:37.582118 21546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:37.602957 21546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:37.603840 21546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:37.604116 21546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:37.604997 21546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:37.606491 21546 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:37.606570 21546 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:37.606709 21546 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:37.606763 21546 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:37.637885 21546 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:36027
I20240501 20:09:37.637951 21672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:36027 every 8 connection(s)
I20240501 20:09:37.639319 21546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:09:37.646847 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 21546
I20240501 20:09:37.647658 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:35605
--local_ip_for_outbound_sockets=127.14.106.197
--tserver_master_addrs=127.14.106.254:36189
--webserver_port=43029
--webserver_interface=127.14.106.197
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:37.653909 21673 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:37.654204 21673 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:37.654992 21673 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:37.656504 20174 ts_manager.cc:194] Re-registered known tserver with Master: 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196:36027)
I20240501 20:09:37.657354 20174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:52995
W20240501 20:09:37.971946 21677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:37.972206 21677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:37.972299 21677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:37.980058 21677 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:37.980173 21677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:37.980290 21677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:09:37.988986 21677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:35605
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=43029
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:37.990154 21677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:37.991782 21677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:38.003927 21683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:38.004719 21682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:38.005496 21685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:38.006435 21677 server_base.cc:1034] running on GCE node
I20240501 20:09:38.007097 21677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:38.008132 21677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:38.009333 21677 hybrid_clock.cc:648] HybridClock initialized: now 1714594178009283 us; error 57 us; skew 500 ppm
I20240501 20:09:38.009646 21677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:38.012006 21677 webserver.cc:460] Webserver started at http://127.14.106.197:43029/ using document root <none> and password file <none>
I20240501 20:09:38.012764 21677 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:38.012908 21677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:38.018193 21677 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:09:38.021129 21691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:38.022497 21677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:38.022686 21677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a7396c6be769436abac350f7d6e86d55"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-pg4x"
I20240501 20:09:38.023550 21677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:38.056051 21677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:38.056864 21677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:38.057143 21677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:38.058075 21677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:38.060010 21698 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 20:09:38.065173 21677 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 20:09:38.065269 21677 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 20:09:38.065402 21677 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 20:09:38.068308 21677 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 20:09:38.068394 21677 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:38.068737 21698 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Bootstrap starting.
I20240501 20:09:38.089000 21698 log.cc:824] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:38.093269 21698 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:09:38.094082 21698 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Bootstrap complete.
I20240501 20:09:38.095489 21698 ts_tablet_manager.cc:1367] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.016s
I20240501 20:09:38.104735 21698 raft_consensus.cc:348] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:38.105962 21698 raft_consensus.cc:729] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.106948 21698 consensus_queue.cc:260] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:38.109086 21698 ts_tablet_manager.cc:1398] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20240501 20:09:38.112834 21677 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:35605
I20240501 20:09:38.112901 21806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:35605 every 8 connection(s)
I20240501 20:09:38.114094 21677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
W20240501 20:09:38.115258 21808 tablet.cc:2369] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:38.116871 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 21677
I20240501 20:09:38.117677 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.198:43715
--local_ip_for_outbound_sockets=127.14.106.198
--tserver_master_addrs=127.14.106.254:36189
--webserver_port=43417
--webserver_interface=127.14.106.198
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:38.127290 21807 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:38.127612 21807 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:38.128659 21807 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:38.131255 20174 ts_manager.cc:194] Re-registered known tserver with Master: a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
I20240501 20:09:38.132329 20174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:38887
W20240501 20:09:38.446902 21810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:38.447160 21810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:38.447292 21810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:38.454553 21810 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:38.454715 21810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:38.454893 21810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.198
I20240501 20:09:38.463599 21810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.198:43715
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.106.198
--webserver_port=43417
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.198
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:38.464826 21810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:38.466491 21810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:38.478808 21816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:38.479584 21815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:38.480397 21818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:38.481209 21810 server_base.cc:1034] running on GCE node
I20240501 20:09:38.481741 21810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:38.482617 21810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:38.483851 21810 hybrid_clock.cc:648] HybridClock initialized: now 1714594178483795 us; error 48 us; skew 500 ppm
I20240501 20:09:38.484220 21810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:38.486658 21810 webserver.cc:460] Webserver started at http://127.14.106.198:43417/ using document root <none> and password file <none>
I20240501 20:09:38.487322 21810 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:38.487454 21810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:38.492201 21810 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 20:09:38.494789 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:38.495996 21810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:38.496145 21810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "133d5e0cbb234be1a5752a784d6d934a"
format_stamp: "Formatted at 2024-05-01 20:09:25 on dist-test-slave-pg4x"
I20240501 20:09:38.496892 21810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:38.536670 21810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:38.537523 21810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:38.537813 21810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:38.538797 21810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:38.540302 21810 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:38.540441 21810 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:38.540602 21810 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:38.540673 21810 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:38.576712 21810 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.198:43715
I20240501 20:09:38.576772 21936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.198:43715 every 8 connection(s)
I20240501 20:09:38.578583 21810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:09:38.579052 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 21810
I20240501 20:09:38.579787 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.199:39669
--local_ip_for_outbound_sockets=127.14.106.199
--tserver_master_addrs=127.14.106.254:36189
--webserver_port=44683
--webserver_interface=127.14.106.199
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:38.593412 21937 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:38.593705 21937 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:38.594414 21937 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:38.595858 20174 ts_manager.cc:194] Re-registered known tserver with Master: 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198:43715)
I20240501 20:09:38.597106 20174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.198:56093
I20240501 20:09:38.659941 21673 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
W20240501 20:09:38.912851 21940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:38.913112 21940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:38.913219 21940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:38.920526 21940 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:38.920639 21940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:38.920745 21940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.199
I20240501 20:09:38.929863 21940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.199:39669
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.106.199
--webserver_port=44683
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.199
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:38.931183 21940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:38.932854 21940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:38.946403 21946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:38.947237 21945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:38.948143 21948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:38.949002 21940 server_base.cc:1034] running on GCE node
I20240501 20:09:38.949708 21940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:38.950665 21940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:38.951874 21940 hybrid_clock.cc:648] HybridClock initialized: now 1714594178951807 us; error 60 us; skew 500 ppm
I20240501 20:09:38.952203 21940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:38.954614 21940 webserver.cc:460] Webserver started at http://127.14.106.199:44683/ using document root <none> and password file <none>
I20240501 20:09:38.955341 21940 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:38.955447 21940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:38.960573 21940 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:09:38.963459 21954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:38.964702 21940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:38.964874 21940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "00e1df10b53f4936b747b168af137e4b"
format_stamp: "Formatted at 2024-05-01 20:09:25 on dist-test-slave-pg4x"
I20240501 20:09:38.965679 21940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:38.981310 21940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:38.982057 21940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:38.982332 21940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:38.983323 21940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:38.985679 21940 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:38.985798 21940 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:38.985930 21940 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:38.986001 21940 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.039458 21940 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.199:39669
I20240501 20:09:39.039489 22066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.199:39669 every 8 connection(s)
I20240501 20:09:39.040831 21940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:39.048779 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 21940
I20240501 20:09:39.049083 14763 test_util.cc:268] Using random seed: 284812887
I20240501 20:09:39.054293 22067 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:09:39.054607 22067 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:39.055337 22067 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:09:39.056706 20173 ts_manager.cc:194] Re-registered known tserver with Master: 00e1df10b53f4936b747b168af137e4b (127.14.106.199:39669)
I20240501 20:09:39.057515 20173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.199:46901
I20240501 20:09:39.065477 14763 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 20:09:39.065881 14763 test_util.cc:268] Using random seed: 284829684
I20240501 20:09:39.080042 14763 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 20:09:39.080425 14763 test_util.cc:268] Using random seed: 284844194
I20240501 20:09:39.119341 14763 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 20:09:39.135439 21807 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
I20240501 20:09:39.153234 14763 test_util.cc:268] Using random seed: 284917027
I20240501 20:09:39.196055 14763 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 20:09:39.208137 14763 test_util.cc:268] Using random seed: 284971926
I20240501 20:09:39.336032 22152 raft_consensus.cc:482] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.336860 22152 raft_consensus.cc:504] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:39.346208 14763 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 20:09:39.353806 22152 leader_election.cc:288] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:39.382438 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4dff67082a6473f994c09821a0d7a3d" candidate_uuid: "a7396c6be769436abac350f7d6e86d55" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed" is_pre_election: true
I20240501 20:09:39.382715 20320 raft_consensus.cc:2454] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7396c6be769436abac350f7d6e86d55 in term 1.
I20240501 20:09:39.383759 21692 leader_election.cc:302] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed, a7396c6be769436abac350f7d6e86d55; no voters:
I20240501 20:09:39.384577 22152 raft_consensus.cc:2781] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:09:39.384740 22152 raft_consensus.cc:482] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:39.384842 22152 raft_consensus.cc:3037] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:39.393244 22152 raft_consensus.cc:504] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:39.394412 22152 leader_election.cc:288] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 2 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:39.395180 20317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4dff67082a6473f994c09821a0d7a3d" candidate_uuid: "a7396c6be769436abac350f7d6e86d55" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:09:39.395375 20317 raft_consensus.cc:3037] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:39.397820 20317 raft_consensus.cc:2454] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7396c6be769436abac350f7d6e86d55 in term 2.
I20240501 20:09:39.400239 21692 leader_election.cc:302] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed, a7396c6be769436abac350f7d6e86d55; no voters:
I20240501 20:09:39.401827 22152 raft_consensus.cc:2781] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:39.410529 22152 raft_consensus.cc:686] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Becoming Leader. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Running, Role: LEADER
I20240501 20:09:39.411239 22152 consensus_queue.cc:237] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } }
I20240501 20:09:39.435500 20173 catalog_manager.cc:5484] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } }
I20240501 20:09:39.599573 21937 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
I20240501 20:09:39.942260 20317 raft_consensus.cc:1261] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:09:39.943153 22152 consensus_queue.cc:1035] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.060221 22067 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
I20240501 20:09:40.580981 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:40.590365 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:40.706135 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:40.743687 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:40.752511 20431 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:40.777671 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:40.947278 21740 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:41.416880 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:41.472676 20431 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:41.529266 21740 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:41.530514 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:41.592989 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:41.606878 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:41.673079 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:42.156075 20581 consensus_queue.cc:237] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:42.168321 20450 raft_consensus.cc:1261] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 53. (index mismatch)
I20240501 20:09:42.178032 22162 consensus_queue.cc:1035] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
W20240501 20:09:42.191864 20517 consensus_peers.cc:485] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 -> Peer 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196:36027): Couldn't send request to peer 79e9c50adfeb44929f5d29524169c6c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e74fb1e0e017408f946a8df8ae304f13. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.194365 21357 raft_consensus.cc:2932] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Committing config change with OpId 1.53: config changed from index -1 to 53, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:42.207587 20450 raft_consensus.cc:2932] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Committing config change with OpId 1.53: config changed from index -1 to 53, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:42.214587 20174 catalog_manager.cc:5484] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 reported cstate change: config changed from index -1 to 53, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:42.369287 20450 consensus_queue.cc:237] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:42.372081 20581 raft_consensus.cc:1261] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240501 20:09:42.372717 22100 consensus_queue.cc:1035] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240501 20:09:42.378945 22100 raft_consensus.cc:2932] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index -1 to 58, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:09:42.379498 20581 raft_consensus.cc:2932] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index -1 to 58, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:09:42.383760 20172 catalog_manager.cc:5484] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index -1 to 58, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
W20240501 20:09:42.409451 20384 consensus_peers.cc:485] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198:43715): Couldn't send request to peer 133d5e0cbb234be1a5752a784d6d934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e414a3678e646c6b5c6c0dad4fac30e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.455657 20450 consensus_queue.cc:237] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:42.463052 20317 raft_consensus.cc:1261] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240501 20:09:42.464483 22112 consensus_queue.cc:1035] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240501 20:09:42.468185 22269 raft_consensus.cc:2932] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:42.472342 20317 raft_consensus.cc:2932] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:42.473857 20174 catalog_manager.cc:5484] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index -1 to 61, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
W20240501 20:09:42.495983 20385 consensus_peers.cc:485] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer 00e1df10b53f4936b747b168af137e4b (127.14.106.199:39669): Couldn't send request to peer 00e1df10b53f4936b747b168af137e4b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee446411ba4e4da2b295080da28d40b9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.656302 20451 consensus_queue.cc:237] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:09:42.681077 20318 raft_consensus.cc:1261] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240501 20:09:42.681959 22113 consensus_queue.cc:1035] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 20:09:42.687795 22113 raft_consensus.cc:2932] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index -1 to 77, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
W20240501 20:09:42.694264 20387 consensus_peers.cc:485] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Couldn't send request to peer a7396c6be769436abac350f7d6e86d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afabe33672c647059a9c125c76906fac. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.698725 22344 ts_tablet_manager.cc:902] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: Initiating tablet copy from peer 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:42.699755 20318 raft_consensus.cc:2932] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index -1 to 77, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:09:42.700223 20172 catalog_manager.cc:5484] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index -1 to 77, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:09:42.721880 22344 tablet_copy_client.cc:308] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35301
I20240501 20:09:42.771764 20601 tablet_copy_service.cc:140] P 2305b26386ac446da2436463b536d296: Received BeginTabletCopySession request for tablet e74fb1e0e017408f946a8df8ae304f13 from peer 79e9c50adfeb44929f5d29524169c6c7 ({username='slave'} at 127.14.106.196:35729)
I20240501 20:09:42.772058 20601 tablet_copy_service.cc:161] P 2305b26386ac446da2436463b536d296: Beginning new tablet copy session on tablet e74fb1e0e017408f946a8df8ae304f13 from peer 79e9c50adfeb44929f5d29524169c6c7 at {username='slave'} at 127.14.106.196:35729: session id = 79e9c50adfeb44929f5d29524169c6c7-e74fb1e0e017408f946a8df8ae304f13
I20240501 20:09:42.776306 20601 tablet_copy_source_session.cc:205] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:42.784492 22344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74fb1e0e017408f946a8df8ae304f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.819269 22363 ts_tablet_manager.cc:902] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:42.834370 22344 tablet_copy_client.cc:783] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:42.834851 22344 tablet_copy_client.cc:647] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:42.847577 22344 tablet_copy_client.cc:523] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:42.858255 22344 tablet_bootstrap.cc:492] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap starting.
I20240501 20:09:42.879575 22363 tablet_copy_client.cc:308] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:09:42.916740 20450 consensus_queue.cc:237] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:42.915840 22344 log.cc:824] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:42.920469 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet 2e414a3678e646c6b5c6c0dad4fac30e from peer 133d5e0cbb234be1a5752a784d6d934a ({username='slave'} at 127.14.106.198:35239)
I20240501 20:09:42.920779 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet 2e414a3678e646c6b5c6c0dad4fac30e from peer 133d5e0cbb234be1a5752a784d6d934a at {username='slave'} at 127.14.106.198:35239: session id = 133d5e0cbb234be1a5752a784d6d934a-2e414a3678e646c6b5c6c0dad4fac30e
I20240501 20:09:42.922521 20578 raft_consensus.cc:1261] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 20:09:42.924055 22113 consensus_queue.cc:1035] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 20:09:42.927124 20471 tablet_copy_source_session.cc:205] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:42.930052 22112 raft_consensus.cc:2932] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:42.933598 22363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e414a3678e646c6b5c6c0dad4fac30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.935487 20578 raft_consensus.cc:2932] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:42.936272 20172 catalog_manager.cc:5484] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index -1 to 79, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:42.968576 22381 ts_tablet_manager.cc:902] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:42.984387 22363 tablet_copy_client.cc:783] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:42.984726 22363 tablet_copy_client.cc:647] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Starting download of 1 WAL segments...
W20240501 20:09:42.989293 20387 consensus_peers.cc:485] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196:36027): Couldn't send request to peer 79e9c50adfeb44929f5d29524169c6c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4c119cb173942819ec01f7f3797c9d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.995463 22363 tablet_copy_client.cc:523] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.002710 22363 tablet_bootstrap.cc:492] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap starting.
I20240501 20:09:43.022372 22381 tablet_copy_client.cc:308] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:09:43.065611 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet ee446411ba4e4da2b295080da28d40b9 from peer 00e1df10b53f4936b747b168af137e4b ({username='slave'} at 127.14.106.199:50261)
I20240501 20:09:43.065838 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet ee446411ba4e4da2b295080da28d40b9 from peer 00e1df10b53f4936b747b168af137e4b at {username='slave'} at 127.14.106.199:50261: session id = 00e1df10b53f4936b747b168af137e4b-ee446411ba4e4da2b295080da28d40b9
I20240501 20:09:43.069105 20471 tablet_copy_source_session.cc:205] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.081110 22381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee446411ba4e4da2b295080da28d40b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.138265 22381 tablet_copy_client.cc:783] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.138849 22381 tablet_copy_client.cc:647] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.150727 22381 tablet_copy_client.cc:523] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.155510 22381 tablet_bootstrap.cc:492] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: Bootstrap starting.
I20240501 20:09:43.160205 22363 log.cc:824] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:43.192898 22381 log.cc:824] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:43.287187 22396 ts_tablet_manager.cc:902] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:43.323083 22381 tablet_bootstrap.cc:492] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.323848 22381 tablet_bootstrap.cc:492] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: Bootstrap complete.
I20240501 20:09:43.324474 22381 ts_tablet_manager.cc:1367] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: Time spent bootstrapping tablet: real 0.169s user 0.136s sys 0.018s
I20240501 20:09:43.333012 22381 raft_consensus.cc:348] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:43.333756 22381 raft_consensus.cc:729] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e1df10b53f4936b747b168af137e4b, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.334810 22381 consensus_queue.cc:260] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:43.343413 22381 ts_tablet_manager.cc:1398] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b: Time spent starting tablet: real 0.019s user 0.009s sys 0.008s
I20240501 20:09:43.349138 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session 00e1df10b53f4936b747b168af137e4b-ee446411ba4e4da2b295080da28d40b9 received from {username='slave'} at 127.14.106.199:50261
I20240501 20:09:43.349309 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session 00e1df10b53f4936b747b168af137e4b-ee446411ba4e4da2b295080da28d40b9 on tablet ee446411ba4e4da2b295080da28d40b9 with peer 00e1df10b53f4936b747b168af137e4b
I20240501 20:09:43.428225 22403 ts_tablet_manager.cc:902] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:43.501129 22363 tablet_bootstrap.cc:492] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=76 ignored=0} inserts{seen=707 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.507354 22396 tablet_copy_client.cc:308] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:09:43.511808 22363 tablet_bootstrap.cc:492] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap complete.
I20240501 20:09:43.513231 22363 ts_tablet_manager.cc:1367] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: Time spent bootstrapping tablet: real 0.511s user 0.146s sys 0.028s
I20240501 20:09:43.514436 22344 tablet_bootstrap.cc:492] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.520970 22363 raft_consensus.cc:348] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:43.521698 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet afabe33672c647059a9c125c76906fac from peer a7396c6be769436abac350f7d6e86d55 ({username='slave'} at 127.14.106.197:43223)
I20240501 20:09:43.521889 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet afabe33672c647059a9c125c76906fac from peer a7396c6be769436abac350f7d6e86d55 at {username='slave'} at 127.14.106.197:43223: session id = a7396c6be769436abac350f7d6e86d55-afabe33672c647059a9c125c76906fac
I20240501 20:09:43.522812 22344 tablet_bootstrap.cc:492] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap complete.
I20240501 20:09:43.524040 22344 ts_tablet_manager.cc:1367] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: Time spent bootstrapping tablet: real 0.666s user 0.138s sys 0.030s
I20240501 20:09:43.525732 20471 tablet_copy_source_session.cc:205] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.528009 22363 raft_consensus.cc:729] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133d5e0cbb234be1a5752a784d6d934a, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.529407 22363 consensus_queue.cc:260] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 76, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:43.543692 22022 raft_consensus.cc:1203] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.66] Dedup: 1.63->[1.64-1.66]
I20240501 20:09:43.547871 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afabe33672c647059a9c125c76906fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.561528 22396 tablet_copy_client.cc:783] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.561777 22344 raft_consensus.cc:348] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:43.562310 22396 tablet_copy_client.cc:647] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.562359 22344 raft_consensus.cc:729] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e9c50adfeb44929f5d29524169c6c7, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.563294 22344 consensus_queue.cc:260] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:43.585619 22344 ts_tablet_manager.cc:1398] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7: Time spent starting tablet: real 0.061s user 0.011s sys 0.003s
I20240501 20:09:43.587118 22396 tablet_copy_client.cc:523] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.588281 20601 tablet_copy_service.cc:342] P 2305b26386ac446da2436463b536d296: Request end of tablet copy session 79e9c50adfeb44929f5d29524169c6c7-e74fb1e0e017408f946a8df8ae304f13 received from {username='slave'} at 127.14.106.196:35729
I20240501 20:09:43.588510 20601 tablet_copy_service.cc:433] P 2305b26386ac446da2436463b536d296: ending tablet copy session 79e9c50adfeb44929f5d29524169c6c7-e74fb1e0e017408f946a8df8ae304f13 on tablet e74fb1e0e017408f946a8df8ae304f13 with peer 79e9c50adfeb44929f5d29524169c6c7
I20240501 20:09:43.596812 22396 tablet_bootstrap.cc:492] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: Bootstrap starting.
I20240501 20:09:43.608008 22363 ts_tablet_manager.cc:1398] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a: Time spent starting tablet: real 0.094s user 0.014s sys 0.000s
I20240501 20:09:43.622774 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session 133d5e0cbb234be1a5752a784d6d934a-2e414a3678e646c6b5c6c0dad4fac30e received from {username='slave'} at 127.14.106.198:35239
I20240501 20:09:43.623031 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session 133d5e0cbb234be1a5752a784d6d934a-2e414a3678e646c6b5c6c0dad4fac30e on tablet 2e414a3678e646c6b5c6c0dad4fac30e with peer 133d5e0cbb234be1a5752a784d6d934a
I20240501 20:09:43.632146 20578 consensus_queue.cc:237] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:43.647814 20451 raft_consensus.cc:1261] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240501 20:09:43.661172 22162 consensus_queue.cc:1035] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240501 20:09:43.675278 22162 raft_consensus.cc:2932] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index -1 to 88, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:09:43.678885 20451 raft_consensus.cc:2932] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index -1 to 88, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:09:43.684324 21892 raft_consensus.cc:1203] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.86] Dedup: 1.76->[1.77-1.86]
I20240501 20:09:43.684978 20172 catalog_manager.cc:5484] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 reported cstate change: config changed from index -1 to 88, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:09:43.758601 22403 tablet_copy_client.cc:308] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:09:43.796339 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet e4c119cb173942819ec01f7f3797c9d7 from peer 79e9c50adfeb44929f5d29524169c6c7 ({username='slave'} at 127.14.106.196:38031)
I20240501 20:09:43.796604 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet e4c119cb173942819ec01f7f3797c9d7 from peer 79e9c50adfeb44929f5d29524169c6c7 at {username='slave'} at 127.14.106.196:38031: session id = 79e9c50adfeb44929f5d29524169c6c7-e4c119cb173942819ec01f7f3797c9d7
I20240501 20:09:43.800086 20471 tablet_copy_source_session.cc:205] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.801304 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c119cb173942819ec01f7f3797c9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.815702 21628 raft_consensus.cc:1203] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.77] Dedup: 1.71->[1.72-1.77]
I20240501 20:09:43.821698 22403 tablet_copy_client.cc:783] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.822019 22403 tablet_copy_client.cc:647] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.824111 22403 tablet_copy_client.cc:523] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.828743 22403 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap starting.
W20240501 20:09:43.886453 20514 consensus_peers.cc:485] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 -> Peer 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198:43715): Couldn't send request to peer 133d5e0cbb234be1a5752a784d6d934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8477d3f66b940538bb74f2cf87a3d47. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:44.149755 20450 consensus_queue.cc:237] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
W20240501 20:09:44.154237 20385 consensus_peers.cc:485] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer 00e1df10b53f4936b747b168af137e4b (127.14.106.199:39669): Couldn't send request to peer 00e1df10b53f4936b747b168af137e4b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9480642e71d46679005d4f55d2e88f8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:44.164429 20316 raft_consensus.cc:1261] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240501 20:09:44.173091 22113 consensus_queue.cc:1035] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240501 20:09:44.186959 22396 tablet_bootstrap.cc:492] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:44.187553 22396 tablet_bootstrap.cc:492] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: Bootstrap complete.
I20240501 20:09:44.187925 22396 ts_tablet_manager.cc:1367] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: Time spent bootstrapping tablet: real 0.591s user 0.186s sys 0.001s
I20240501 20:09:44.191526 22396 raft_consensus.cc:348] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:09:44.191874 22396 raft_consensus.cc:729] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.193640 22396 consensus_queue.cc:260] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:09:44.195112 22396 ts_tablet_manager.cc:1398] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:44.195693 21761 raft_consensus.cc:1203] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.76->[1.77-1.98] Dedup: 1.91->[1.92-1.98]
I20240501 20:09:44.202723 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session a7396c6be769436abac350f7d6e86d55-afabe33672c647059a9c125c76906fac received from {username='slave'} at 127.14.106.197:43223
I20240501 20:09:44.202867 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session a7396c6be769436abac350f7d6e86d55-afabe33672c647059a9c125c76906fac on tablet afabe33672c647059a9c125c76906fac with peer a7396c6be769436abac350f7d6e86d55
I20240501 20:09:44.204336 22403 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=719 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:44.204824 22403 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap complete.
I20240501 20:09:44.205142 22403 ts_tablet_manager.cc:1367] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: Time spent bootstrapping tablet: real 0.376s user 0.131s sys 0.022s
I20240501 20:09:44.207592 22403 raft_consensus.cc:348] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:44.207898 22403 raft_consensus.cc:729] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e9c50adfeb44929f5d29524169c6c7, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.208235 22403 consensus_queue.cc:260] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:44.208969 22403 ts_tablet_manager.cc:1398] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:09:44.211206 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session 79e9c50adfeb44929f5d29524169c6c7-e4c119cb173942819ec01f7f3797c9d7 received from {username='slave'} at 127.14.106.196:38031
I20240501 20:09:44.211331 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session 79e9c50adfeb44929f5d29524169c6c7-e4c119cb173942819ec01f7f3797c9d7 on tablet e4c119cb173942819ec01f7f3797c9d7 with peer 79e9c50adfeb44929f5d29524169c6c7
I20240501 20:09:44.378508 22113 raft_consensus.cc:2932] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index -1 to 123, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:44.394703 22471 ts_tablet_manager.cc:902] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: Initiating tablet copy from peer 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:44.426383 20316 raft_consensus.cc:2932] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index -1 to 123, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:44.433367 20174 catalog_manager.cc:5484] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: config changed from index -1 to 123, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:44.537247 21627 raft_consensus.cc:1203] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.78->[1.79-1.98] Dedup: 1.88->[1.89-1.98]
W20240501 20:09:44.597332 20517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:36027, user_credentials={real_user=slave}} blocked reactor thread for 69521.7us
I20240501 20:09:44.722795 22483 ts_tablet_manager.cc:902] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:44.751102 22483 tablet_copy_client.cc:308] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:09:44.752625 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet f9480642e71d46679005d4f55d2e88f8 from peer 00e1df10b53f4936b747b168af137e4b ({username='slave'} at 127.14.106.199:50261)
I20240501 20:09:44.752787 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet f9480642e71d46679005d4f55d2e88f8 from peer 00e1df10b53f4936b747b168af137e4b at {username='slave'} at 127.14.106.199:50261: session id = 00e1df10b53f4936b747b168af137e4b-f9480642e71d46679005d4f55d2e88f8
I20240501 20:09:44.755578 20471 tablet_copy_source_session.cc:205] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.757128 22483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9480642e71d46679005d4f55d2e88f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.773217 22483 tablet_copy_client.cc:783] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.773523 22483 tablet_copy_client.cc:647] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.779762 22471 tablet_copy_client.cc:308] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35301
I20240501 20:09:44.788628 22483 tablet_copy_client.cc:523] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.795856 22483 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: Bootstrap starting.
I20240501 20:09:45.136785 20601 tablet_copy_service.cc:140] P 2305b26386ac446da2436463b536d296: Received BeginTabletCopySession request for tablet e8477d3f66b940538bb74f2cf87a3d47 from peer 133d5e0cbb234be1a5752a784d6d934a ({username='slave'} at 127.14.106.198:36805)
I20240501 20:09:45.136967 20601 tablet_copy_service.cc:161] P 2305b26386ac446da2436463b536d296: Beginning new tablet copy session on tablet e8477d3f66b940538bb74f2cf87a3d47 from peer 133d5e0cbb234be1a5752a784d6d934a at {username='slave'} at 127.14.106.198:36805: session id = 133d5e0cbb234be1a5752a784d6d934a-e8477d3f66b940538bb74f2cf87a3d47
I20240501 20:09:45.139925 20601 tablet_copy_source_session.cc:205] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:45.154276 22471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8477d3f66b940538bb74f2cf87a3d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.163867 22471 tablet_copy_client.cc:783] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:45.164352 22471 tablet_copy_client.cc:647] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:45.186153 22471 tablet_copy_client.cc:523] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:45.192559 22471 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap starting.
I20240501 20:09:45.202342 22483 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=131 overwritten=0 applied=131 ignored=0} inserts{seen=803 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:45.202960 22483 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: Bootstrap complete.
I20240501 20:09:45.203328 22483 ts_tablet_manager.cc:1367] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: Time spent bootstrapping tablet: real 0.408s user 0.216s sys 0.029s
I20240501 20:09:45.204397 22483 raft_consensus.cc:348] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:45.204702 22483 raft_consensus.cc:729] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e1df10b53f4936b747b168af137e4b, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.204994 22483 consensus_queue.cc:260] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 131, Last appended: 1.131, Last appended by leader: 131, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:45.223227 22483 ts_tablet_manager.cc:1398] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b: Time spent starting tablet: real 0.020s user 0.003s sys 0.005s
I20240501 20:09:45.224385 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session 00e1df10b53f4936b747b168af137e4b-f9480642e71d46679005d4f55d2e88f8 received from {username='slave'} at 127.14.106.199:50261
I20240501 20:09:45.224535 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session 00e1df10b53f4936b747b168af137e4b-f9480642e71d46679005d4f55d2e88f8 on tablet f9480642e71d46679005d4f55d2e88f8 with peer 00e1df10b53f4936b747b168af137e4b
I20240501 20:09:45.650022 22022 raft_consensus.cc:1203] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.122->[1.123-1.144] Dedup: 1.131->[1.132-1.144]
I20240501 20:09:45.806044 22471 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap replayed 1/1 log segments. Stats: ops{read=103 overwritten=0 applied=103 ignored=0} inserts{seen=788 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:45.806519 22471 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap complete.
I20240501 20:09:45.806844 22471 ts_tablet_manager.cc:1367] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: Time spent bootstrapping tablet: real 0.614s user 0.152s sys 0.017s
I20240501 20:09:45.808030 22471 raft_consensus.cc:348] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:45.808270 22471 raft_consensus.cc:729] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133d5e0cbb234be1a5752a784d6d934a, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.815186 22471 consensus_queue.cc:260] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:45.830740 22471 ts_tablet_manager.cc:1398] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20240501 20:09:45.836462 20601 tablet_copy_service.cc:342] P 2305b26386ac446da2436463b536d296: Request end of tablet copy session 133d5e0cbb234be1a5752a784d6d934a-e8477d3f66b940538bb74f2cf87a3d47 received from {username='slave'} at 127.14.106.198:36805
I20240501 20:09:45.836625 20601 tablet_copy_service.cc:433] P 2305b26386ac446da2436463b536d296: ending tablet copy session 133d5e0cbb234be1a5752a784d6d934a-e8477d3f66b940538bb74f2cf87a3d47 on tablet e8477d3f66b940538bb74f2cf87a3d47 with peer 133d5e0cbb234be1a5752a784d6d934a
I20240501 20:09:45.991089 21892 raft_consensus.cc:1203] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.87->[1.88-1.120] Dedup: 1.103->[1.104-1.120]
I20240501 20:09:46.097885 20577 consensus_queue.cc:237] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:09:46.116720 20317 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 20:09:46.117337 22513 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240501 20:09:46.119625 22513 raft_consensus.cc:2932] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index -1 to 98, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:09:46.123652 20320 raft_consensus.cc:2932] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:09:46.134419 20174 catalog_manager.cc:5484] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: config changed from index -1 to 98, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
W20240501 20:09:46.412894 20517 consensus_peers.cc:485] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 -> Peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Couldn't send request to peer a7396c6be769436abac350f7d6e86d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a45309550b449c1bf0ba233c24140ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:46.666036 22555 ts_tablet_manager.cc:902] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Initiating tablet copy from peer 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:46.698836 22555 tablet_copy_client.cc:308] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35301
I20240501 20:09:46.736064 20579 consensus_queue.cc:237] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 1.116, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
W20240501 20:09:46.739946 20517 consensus_peers.cc:485] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 -> Peer 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196:36027): Couldn't send request to peer 79e9c50adfeb44929f5d29524169c6c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86d6707287cc48169ae7a548c1c01597. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:46.745182 20317 raft_consensus.cc:1261] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240501 20:09:46.750520 20601 tablet_copy_service.cc:140] P 2305b26386ac446da2436463b536d296: Received BeginTabletCopySession request for tablet 5a45309550b449c1bf0ba233c24140ed from peer a7396c6be769436abac350f7d6e86d55 ({username='slave'} at 127.14.106.197:39287)
I20240501 20:09:46.750703 20601 tablet_copy_service.cc:161] P 2305b26386ac446da2436463b536d296: Beginning new tablet copy session on tablet 5a45309550b449c1bf0ba233c24140ed from peer a7396c6be769436abac350f7d6e86d55 at {username='slave'} at 127.14.106.197:39287: session id = a7396c6be769436abac350f7d6e86d55-5a45309550b449c1bf0ba233c24140ed
I20240501 20:09:46.751008 22123 consensus_queue.cc:1035] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 115, Time since last communication: 0.000s
I20240501 20:09:46.753736 20601 tablet_copy_source_session.cc:205] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:46.755993 22571 raft_consensus.cc:2932] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Committing config change with OpId 1.117: config changed from index -1 to 117, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:46.758080 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a45309550b449c1bf0ba233c24140ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.760643 20317 raft_consensus.cc:2932] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index -1 to 117, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:46.765388 22555 tablet_copy_client.cc:783] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:46.765646 22555 tablet_copy_client.cc:647] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:46.772516 20173 catalog_manager.cc:5484] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 reported cstate change: config changed from index -1 to 117, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:46.775045 22555 tablet_copy_client.cc:523] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:46.779654 22555 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Bootstrap starting.
I20240501 20:09:47.206089 22555 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=107 overwritten=0 applied=107 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:47.209146 22555 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Bootstrap complete.
I20240501 20:09:47.209515 22555 ts_tablet_manager.cc:1367] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Time spent bootstrapping tablet: real 0.430s user 0.164s sys 0.007s
I20240501 20:09:47.210376 22555 raft_consensus.cc:348] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:09:47.210669 22555 raft_consensus.cc:729] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.210974 22555 consensus_queue.cc:260] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:09:47.214172 22555 ts_tablet_manager.cc:1398] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 20:09:47.215679 20601 tablet_copy_service.cc:342] P 2305b26386ac446da2436463b536d296: Request end of tablet copy session a7396c6be769436abac350f7d6e86d55-5a45309550b449c1bf0ba233c24140ed received from {username='slave'} at 127.14.106.197:39287
I20240501 20:09:47.215837 20601 tablet_copy_service.cc:433] P 2305b26386ac446da2436463b536d296: ending tablet copy session a7396c6be769436abac350f7d6e86d55-5a45309550b449c1bf0ba233c24140ed on tablet 5a45309550b449c1bf0ba233c24140ed with peer a7396c6be769436abac350f7d6e86d55
I20240501 20:09:47.287807 22589 ts_tablet_manager.cc:902] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: Initiating tablet copy from peer 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:09:47.397843 22589 tablet_copy_client.cc:308] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35301
I20240501 20:09:47.398476 20601 tablet_copy_service.cc:140] P 2305b26386ac446da2436463b536d296: Received BeginTabletCopySession request for tablet 86d6707287cc48169ae7a548c1c01597 from peer 79e9c50adfeb44929f5d29524169c6c7 ({username='slave'} at 127.14.106.196:35729)
I20240501 20:09:47.398710 20601 tablet_copy_service.cc:161] P 2305b26386ac446da2436463b536d296: Beginning new tablet copy session on tablet 86d6707287cc48169ae7a548c1c01597 from peer 79e9c50adfeb44929f5d29524169c6c7 at {username='slave'} at 127.14.106.196:35729: session id = 79e9c50adfeb44929f5d29524169c6c7-86d6707287cc48169ae7a548c1c01597
I20240501 20:09:47.401590 20601 tablet_copy_source_session.cc:205] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:47.402976 22589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d6707287cc48169ae7a548c1c01597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.415236 22589 tablet_copy_client.cc:783] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:47.415625 22589 tablet_copy_client.cc:647] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:47.420928 22589 tablet_copy_client.cc:523] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:47.437355 22589 tablet_bootstrap.cc:492] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap starting.
I20240501 20:09:47.548573 20450 consensus_queue.cc:237] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:47.553974 20577 raft_consensus.cc:1261] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240501 20:09:47.555282 21541 consensus_queue.cc:1035] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
W20240501 20:09:47.591363 20384 consensus_peers.cc:485] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198:43715): Couldn't send request to peer 133d5e0cbb234be1a5752a784d6d934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01bf239579d84c07adff90ca8f87824d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:47.605041 21760 raft_consensus.cc:1203] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.97->[1.98-1.124] Dedup: 1.107->[1.108-1.124]
I20240501 20:09:47.620342 21541 raft_consensus.cc:2932] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index -1 to 159, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
W20240501 20:09:47.652385 20385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35301, user_credentials={real_user=slave}} blocked reactor thread for 50574.8us
I20240501 20:09:47.658095 20173 catalog_manager.cc:5484] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index -1 to 159, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:09:47.658291 20581 raft_consensus.cc:2932] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index -1 to 159, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:09:48.000630 22589 tablet_bootstrap.cc:492] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=133 overwritten=0 applied=133 ignored=0} inserts{seen=818 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:48.001205 22589 tablet_bootstrap.cc:492] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap complete.
I20240501 20:09:48.001580 22589 ts_tablet_manager.cc:1367] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: Time spent bootstrapping tablet: real 0.564s user 0.269s sys 0.132s
I20240501 20:09:48.002295 22589 raft_consensus.cc:348] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:48.002493 22589 raft_consensus.cc:729] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e9c50adfeb44929f5d29524169c6c7, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.002816 22589 consensus_queue.cc:260] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 133, Last appended: 1.133, Last appended by leader: 133, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:48.028782 22589 ts_tablet_manager.cc:1398] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7: Time spent starting tablet: real 0.027s user 0.008s sys 0.013s
I20240501 20:09:48.031096 20601 tablet_copy_service.cc:342] P 2305b26386ac446da2436463b536d296: Request end of tablet copy session 79e9c50adfeb44929f5d29524169c6c7-86d6707287cc48169ae7a548c1c01597 received from {username='slave'} at 127.14.106.196:35729
I20240501 20:09:48.031229 20601 tablet_copy_service.cc:433] P 2305b26386ac446da2436463b536d296: ending tablet copy session 79e9c50adfeb44929f5d29524169c6c7-86d6707287cc48169ae7a548c1c01597 on tablet 86d6707287cc48169ae7a548c1c01597 with peer 79e9c50adfeb44929f5d29524169c6c7
I20240501 20:09:48.037974 22627 ts_tablet_manager.cc:902] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:48.187777 22627 tablet_copy_client.cc:308] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:09:48.194366 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet 01bf239579d84c07adff90ca8f87824d from peer 133d5e0cbb234be1a5752a784d6d934a ({username='slave'} at 127.14.106.198:35239)
I20240501 20:09:48.194546 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet 01bf239579d84c07adff90ca8f87824d from peer 133d5e0cbb234be1a5752a784d6d934a at {username='slave'} at 127.14.106.198:35239: session id = 133d5e0cbb234be1a5752a784d6d934a-01bf239579d84c07adff90ca8f87824d
I20240501 20:09:48.197654 20471 tablet_copy_source_session.cc:205] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:48.199326 22627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bf239579d84c07adff90ca8f87824d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.208621 22627 tablet_copy_client.cc:783] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:48.208850 22627 tablet_copy_client.cc:647] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:48.211731 22627 tablet_copy_client.cc:523] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:48.221279 22627 tablet_bootstrap.cc:492] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap starting.
I20240501 20:09:48.283371 20450 consensus_queue.cc:237] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 1.157, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:48.286243 20579 raft_consensus.cc:1261] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240501 20:09:48.287004 22113 consensus_queue.cc:1035] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 156, Time since last communication: 0.000s
W20240501 20:09:48.287351 20385 consensus_peers.cc:485] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer 00e1df10b53f4936b747b168af137e4b (127.14.106.199:39669): Couldn't send request to peer 00e1df10b53f4936b747b168af137e4b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62a6957734c44303929840f208f5d1d3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:48.293294 20579 raft_consensus.cc:2932] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index -1 to 158, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:48.300475 20174 catalog_manager.cc:5484] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 reported cstate change: config changed from index -1 to 158, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:48.301769 22112 raft_consensus.cc:2932] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.158: config changed from index -1 to 158, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:48.386234 21626 raft_consensus.cc:1203] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.116->[1.117-1.147] Dedup: 1.133->[1.134-1.147]
I20240501 20:09:48.523417 22627 tablet_bootstrap.cc:492] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap replayed 1/1 log segments. Stats: ops{read=162 overwritten=0 applied=162 ignored=0} inserts{seen=793 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:48.524519 22627 tablet_bootstrap.cc:492] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap complete.
I20240501 20:09:48.524951 22627 ts_tablet_manager.cc:1367] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: Time spent bootstrapping tablet: real 0.304s user 0.231s sys 0.035s
I20240501 20:09:48.525964 22627 raft_consensus.cc:348] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:48.526753 22627 raft_consensus.cc:729] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133d5e0cbb234be1a5752a784d6d934a, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.527156 22627 consensus_queue.cc:260] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 162, Last appended: 1.162, Last appended by leader: 162, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:09:48.534965 22627 ts_tablet_manager.cc:1398] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20240501 20:09:48.597493 21892 raft_consensus.cc:1203] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.158->[1.159-1.166] Dedup: 1.162->[1.163-1.166]
I20240501 20:09:48.682384 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session 133d5e0cbb234be1a5752a784d6d934a-01bf239579d84c07adff90ca8f87824d received from {username='slave'} at 127.14.106.198:35239
I20240501 20:09:48.682569 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session 133d5e0cbb234be1a5752a784d6d934a-01bf239579d84c07adff90ca8f87824d on tablet 01bf239579d84c07adff90ca8f87824d with peer 133d5e0cbb234be1a5752a784d6d934a
I20240501 20:09:48.709542 22668 ts_tablet_manager.cc:902] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:09:48.762190 22668 tablet_copy_client.cc:308] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
W20240501 20:09:48.850921 20384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:36237, user_credentials={real_user=slave}} blocked reactor thread for 110467us
I20240501 20:09:48.854013 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet 62a6957734c44303929840f208f5d1d3 from peer 00e1df10b53f4936b747b168af137e4b ({username='slave'} at 127.14.106.199:50261)
I20240501 20:09:48.854204 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet 62a6957734c44303929840f208f5d1d3 from peer 00e1df10b53f4936b747b168af137e4b at {username='slave'} at 127.14.106.199:50261: session id = 00e1df10b53f4936b747b168af137e4b-62a6957734c44303929840f208f5d1d3
I20240501 20:09:48.859956 20471 tablet_copy_source_session.cc:205] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:48.867501 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a6957734c44303929840f208f5d1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.878369 22668 tablet_copy_client.cc:783] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:48.878867 22668 tablet_copy_client.cc:647] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Starting download of 1 WAL segments...
W20240501 20:09:48.902797 21692 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:36237, user_credentials={real_user=slave}} blocked reactor thread for 117044us
I20240501 20:09:48.906143 22668 tablet_copy_client.cc:523] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:48.911620 22668 tablet_bootstrap.cc:492] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: Bootstrap starting.
I20240501 20:09:49.314952 22668 tablet_bootstrap.cc:492] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=165 overwritten=0 applied=165 ignored=0} inserts{seen=850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:49.315793 22668 tablet_bootstrap.cc:492] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: Bootstrap complete.
I20240501 20:09:49.316156 22668 ts_tablet_manager.cc:1367] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: Time spent bootstrapping tablet: real 0.405s user 0.232s sys 0.016s
I20240501 20:09:49.316833 22668 raft_consensus.cc:348] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:49.317095 22668 raft_consensus.cc:729] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e1df10b53f4936b747b168af137e4b, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.317363 22668 consensus_queue.cc:260] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 165, Last appended: 1.165, Last appended by leader: 165, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:49.324338 22668 ts_tablet_manager.cc:1398] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240501 20:09:49.340251 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session 00e1df10b53f4936b747b168af137e4b-62a6957734c44303929840f208f5d1d3 received from {username='slave'} at 127.14.106.199:50261
I20240501 20:09:49.340425 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session 00e1df10b53f4936b747b168af137e4b-62a6957734c44303929840f208f5d1d3 on tablet 62a6957734c44303929840f208f5d1d3 with peer 00e1df10b53f4936b747b168af137e4b
I20240501 20:09:49.421932 21760 consensus_queue.cc:237] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:49.446796 20318 raft_consensus.cc:1261] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:09:49.448144 22673 consensus_queue.cc:1035] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 20:09:49.495129 21693 consensus_peers.cc:485] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 -> Peer 00e1df10b53f4936b747b168af137e4b (127.14.106.199:39669): Couldn't send request to peer 00e1df10b53f4936b747b168af137e4b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4dff67082a6473f994c09821a0d7a3d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:49.524430 22673 raft_consensus.cc:2932] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:49.525163 20316 raft_consensus.cc:2932] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:49.537704 20174 catalog_manager.cc:5484] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: config changed from index -1 to 3, VOTER 00e1df10b53f4936b747b168af137e4b (127.14.106.199) added. New cstate: current_term: 2 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:49.762418 22021 raft_consensus.cc:1203] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.157->[1.158-1.174] Dedup: 1.165->[1.166-1.174]
I20240501 20:09:49.992211 22719 ts_tablet_manager.cc:902] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: Initiating tablet copy from peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
I20240501 20:09:50.142807 22719 tablet_copy_client.cc:308] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.197:35605
I20240501 20:09:50.233037 21781 tablet_copy_service.cc:140] P a7396c6be769436abac350f7d6e86d55: Received BeginTabletCopySession request for tablet e4dff67082a6473f994c09821a0d7a3d from peer 00e1df10b53f4936b747b168af137e4b ({username='slave'} at 127.14.106.199:33675)
I20240501 20:09:50.233335 21781 tablet_copy_service.cc:161] P a7396c6be769436abac350f7d6e86d55: Beginning new tablet copy session on tablet e4dff67082a6473f994c09821a0d7a3d from peer 00e1df10b53f4936b747b168af137e4b at {username='slave'} at 127.14.106.199:33675: session id = 00e1df10b53f4936b747b168af137e4b-e4dff67082a6473f994c09821a0d7a3d
I20240501 20:09:50.237504 21781 tablet_copy_source_session.cc:205] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:50.241521 22719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4dff67082a6473f994c09821a0d7a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.248199 22719 tablet_copy_client.cc:783] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:50.248522 22719 tablet_copy_client.cc:647] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:50.250993 22719 tablet_copy_client.cc:523] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:50.254457 22719 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: Bootstrap starting.
I20240501 20:09:50.281585 22719 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:50.282203 22719 tablet_bootstrap.cc:492] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: Bootstrap complete.
I20240501 20:09:50.282539 22719 ts_tablet_manager.cc:1367] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.002s
I20240501 20:09:50.287950 22719 raft_consensus.cc:348] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:50.288271 22719 raft_consensus.cc:729] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e1df10b53f4936b747b168af137e4b, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.288566 22719 consensus_queue.cc:260] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:50.299155 20318 consensus_queue.cc:237] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.240, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:50.308291 20581 raft_consensus.cc:1261] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9458da0b406db99095564f9cf2ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20240501 20:09:50.314402 22587 consensus_queue.cc:1035] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20240501 20:09:50.317674 22587 raft_consensus.cc:2932] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Committing config change with OpId 1.241: config changed from index -1 to 241, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:50.322237 20577 raft_consensus.cc:2932] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index -1 to 241, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:50.333264 20173 catalog_manager.cc:5484] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: config changed from index -1 to 241, VOTER 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "4f8d9458da0b406db99095564f9cf2ed" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:50.345425 22719 ts_tablet_manager.cc:1398] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b: Time spent starting tablet: real 0.059s user 0.004s sys 0.026s
I20240501 20:09:50.351740 21781 tablet_copy_service.cc:342] P a7396c6be769436abac350f7d6e86d55: Request end of tablet copy session 00e1df10b53f4936b747b168af137e4b-e4dff67082a6473f994c09821a0d7a3d received from {username='slave'} at 127.14.106.199:33675
I20240501 20:09:50.351954 21781 tablet_copy_service.cc:433] P a7396c6be769436abac350f7d6e86d55: ending tablet copy session 00e1df10b53f4936b747b168af137e4b-e4dff67082a6473f994c09821a0d7a3d on tablet e4dff67082a6473f994c09821a0d7a3d with peer 00e1df10b53f4936b747b168af137e4b
W20240501 20:09:50.446965 20256 consensus_peers.cc:485] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed -> Peer 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196:36027): Couldn't send request to peer 79e9c50adfeb44929f5d29524169c6c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ff0eebf61824be1aff2cf5c43f45d78. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:50.473179 22020 raft_consensus.cc:1203] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240501 20:09:50.926726 22761 ts_tablet_manager.cc:902] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: Initiating tablet copy from peer 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:09:51.011154 22761 tablet_copy_client.cc:308] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:36237
I20240501 20:09:51.227311 20340 tablet_copy_service.cc:140] P 4f8d9458da0b406db99095564f9cf2ed: Received BeginTabletCopySession request for tablet 7ff0eebf61824be1aff2cf5c43f45d78 from peer 79e9c50adfeb44929f5d29524169c6c7 ({username='slave'} at 127.14.106.196:51621)
I20240501 20:09:51.227921 20340 tablet_copy_service.cc:161] P 4f8d9458da0b406db99095564f9cf2ed: Beginning new tablet copy session on tablet 7ff0eebf61824be1aff2cf5c43f45d78 from peer 79e9c50adfeb44929f5d29524169c6c7 at {username='slave'} at 127.14.106.196:51621: session id = 79e9c50adfeb44929f5d29524169c6c7-7ff0eebf61824be1aff2cf5c43f45d78
I20240501 20:09:51.233472 20340 tablet_copy_source_session.cc:205] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:51.245716 22761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ff0eebf61824be1aff2cf5c43f45d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.251112 22761 tablet_copy_client.cc:783] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:51.251384 22761 tablet_copy_client.cc:647] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:51.264032 22761 tablet_copy_client.cc:523] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:51.278492 22761 tablet_bootstrap.cc:492] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap starting.
W20240501 20:09:51.739951 20514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 56379.3us
I20240501 20:09:52.145619 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:52.390817 21740 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:52.424050 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:52.447578 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:52.463176 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:52.499694 20431 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:52.592787 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:09:52.739604 20384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 101348us
I20240501 20:09:53.036697 22761 tablet_bootstrap.cc:492] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=252 overwritten=0 applied=252 ignored=0} inserts{seen=937 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:53.037300 22761 tablet_bootstrap.cc:492] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: Bootstrap complete.
I20240501 20:09:53.037642 22761 ts_tablet_manager.cc:1367] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: Time spent bootstrapping tablet: real 1.759s user 0.338s sys 0.016s
I20240501 20:09:53.038910 22761 raft_consensus.cc:348] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:53.039175 22761 raft_consensus.cc:729] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e9c50adfeb44929f5d29524169c6c7, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.039433 22761 consensus_queue.cc:260] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 252, Last appended: 1.252, Last appended by leader: 252, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:53.046252 22761 ts_tablet_manager.cc:1398] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240501 20:09:53.047449 20340 tablet_copy_service.cc:342] P 4f8d9458da0b406db99095564f9cf2ed: Request end of tablet copy session 79e9c50adfeb44929f5d29524169c6c7-7ff0eebf61824be1aff2cf5c43f45d78 received from {username='slave'} at 127.14.106.196:51621
I20240501 20:09:53.047609 20340 tablet_copy_service.cc:433] P 4f8d9458da0b406db99095564f9cf2ed: ending tablet copy session 79e9c50adfeb44929f5d29524169c6c7-7ff0eebf61824be1aff2cf5c43f45d78 on tablet 7ff0eebf61824be1aff2cf5c43f45d78 with peer 79e9c50adfeb44929f5d29524169c6c7
I20240501 20:09:53.309511 21627 raft_consensus.cc:1203] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.240->[1.241-1.292] Dedup: 1.252->[1.253-1.292]
W20240501 20:09:53.433311 20384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:36237, user_credentials={real_user=slave}} blocked reactor thread for 58908.7us
I20240501 20:09:53.595466 20581 consensus_queue.cc:237] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 1.228, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:53.606446 21626 raft_consensus.cc:1261] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 228. Preceding OpId from leader: term: 1 index: 229. (index mismatch)
I20240501 20:09:53.613669 22813 consensus_queue.cc:1035] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 227, Time since last communication: 0.000s
I20240501 20:09:53.623340 22813 raft_consensus.cc:2932] T 86d6707287cc48169ae7a548c1c01597 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Committing config change with OpId 1.229: config changed from index 117 to 229, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:53.624984 21626 raft_consensus.cc:2932] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Committing config change with OpId 1.229: config changed from index 117 to 229, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:53.639367 20173 catalog_manager.cc:5484] T 86d6707287cc48169ae7a548c1c01597 P 79e9c50adfeb44929f5d29524169c6c7 reported cstate change: config changed from index 117 to 229, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:53.716104 20298 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d6707287cc48169ae7a548c1c01597 with delete_type TABLET_DATA_TOMBSTONED (TS 4f8d9458da0b406db99095564f9cf2ed not found in new config with opid_index 229) from {username='slave'} at 127.0.0.1:34004
I20240501 20:09:53.730854 22860 tablet_replica.cc:330] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: stopping tablet replica
I20240501 20:09:53.731365 22860 raft_consensus.cc:2229] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:53.731709 22860 raft_consensus.cc:2258] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:53.737924 22860 ts_tablet_manager.cc:1875] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:53.746764 22860 ts_tablet_manager.cc:1888] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.227
I20240501 20:09:53.747030 22860 log.cc:1196] T 86d6707287cc48169ae7a548c1c01597 P 4f8d9458da0b406db99095564f9cf2ed: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/86d6707287cc48169ae7a548c1c01597
I20240501 20:09:53.748109 20163 catalog_manager.cc:4830] TS 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237): tablet 86d6707287cc48169ae7a548c1c01597 (table rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]) successfully deleted
I20240501 20:09:54.773212 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:54.825966 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:54.845657 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:54.928699 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:54.940939 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:54.975091 20431 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:55.032831 21740 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:56.221707 20317 consensus_queue.cc:237] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 1.364, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:09:56.223373 21627 raft_consensus.cc:1261] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9458da0b406db99095564f9cf2ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 364. Preceding OpId from leader: term: 1 index: 365. (index mismatch)
I20240501 20:09:56.232901 22619 consensus_queue.cc:1035] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20240501 20:09:56.235399 22619 raft_consensus.cc:2932] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Committing config change with OpId 1.365: config changed from index 241 to 365, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:56.239944 21627 raft_consensus.cc:2932] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Committing config change with OpId 1.365: config changed from index 241 to 365, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:56.241178 20173 catalog_manager.cc:5484] T 7ff0eebf61824be1aff2cf5c43f45d78 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: config changed from index 241 to 365, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "4f8d9458da0b406db99095564f9cf2ed" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:09:56.330153 20560 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ff0eebf61824be1aff2cf5c43f45d78 with delete_type TABLET_DATA_TOMBSTONED (TS 2305b26386ac446da2436463b536d296 not found in new config with opid_index 365) from {username='slave'} at 127.0.0.1:49878
I20240501 20:09:56.331817 22962 tablet_replica.cc:330] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: stopping tablet replica
I20240501 20:09:56.336167 22962 raft_consensus.cc:2229] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:56.336891 22962 pending_rounds.cc:62] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: Trying to abort 1 pending ops.
I20240501 20:09:56.337291 22962 pending_rounds.cc:68] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: Aborting op as it isn't in flight: id { term: 1 index: 364 } timestamp: 7022977827565756416 op_type: WRITE_OP write_request { tablet_id: "7ff0eebf61824be1aff2cf5c43f45d78" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\324w@\003\326V|\026\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a849b4467db4e8a7" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977827502407680 authz_token { token_data: "\010\257\305\312\261\006\"3\n\005slave\022*\n da3676c5753a4bcc8f3d75f8dec89f70\020\001\030\001 \001(\001" signature: "<redacted>""\020\266H;\'C|=\242\377\243\222D&I\301 \216\347=\341\273G\004\237\212\277\317\2369?n\177\227\240\340oL\323\311b\250O\000-lE\315\0064\3636+\004\253)RQo\305\315\006=z" signing_key_seq_num: 0 } } request_id { client_id: "979e7b4a3f654aada0e0b063e747bb65" seq_no: 1047 first_incomplete_seq_no: 1047 attempt_no: 0 }
I20240501 20:09:56.342804 22962 raft_consensus.cc:2258] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:56.345782 22962 ts_tablet_manager.cc:1875] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:56.366652 22962 ts_tablet_manager.cc:1888] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.364
I20240501 20:09:56.367803 22962 log.cc:1196] T 7ff0eebf61824be1aff2cf5c43f45d78 P 2305b26386ac446da2436463b536d296: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/7ff0eebf61824be1aff2cf5c43f45d78
I20240501 20:09:56.372525 20164 catalog_manager.cc:4830] TS 2305b26386ac446da2436463b536d296 (127.14.106.195:35301): tablet 7ff0eebf61824be1aff2cf5c43f45d78 (table rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]) successfully deleted
I20240501 20:09:57.359361 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:57.397238 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:57.408746 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:09:57.463013 20385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:39669, user_credentials={real_user=slave}} blocked reactor thread for 61116.5us
I20240501 20:09:57.523090 20431 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:57.562712 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:57.698043 21740 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:57.698153 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:58.190848 20450 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62a6957734c44303929840f208f5d1d3"
dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41578
I20240501 20:09:58.191141 20450 raft_consensus.cc:595] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:58.195575 23028 raft_consensus.cc:979] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: : Instructing follower 2305b26386ac446da2436463b536d296 to start an election
I20240501 20:09:58.195745 23028 raft_consensus.cc:1067] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Signalling peer 2305b26386ac446da2436463b536d296 to start an election
I20240501 20:09:58.201282 20577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62a6957734c44303929840f208f5d1d3"
dest_uuid: "2305b26386ac446da2436463b536d296"
from {username='slave'} at 127.14.106.194:44337
I20240501 20:09:58.201593 20577 raft_consensus.cc:482] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:58.201719 20577 raft_consensus.cc:3037] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:58.204362 20577 raft_consensus.cc:504] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:58.205158 20577 leader_election.cc:288] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 2 election: Requested vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937), 00e1df10b53f4936b747b168af137e4b (127.14.106.199:39669)
W20240501 20:09:58.216445 20408 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40850: Service unavailable: leader transfer in progress
I20240501 20:09:58.222741 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a6957734c44303929840f208f5d1d3" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 2 candidate_status { last_received { term: 1 index: 300 } } ignore_live_leader: true dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
I20240501 20:09:58.222952 20451 raft_consensus.cc:3032] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:58.223079 20451 raft_consensus.cc:729] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:09:58.223373 20451 consensus_queue.cc:260] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 1.300, Last appended by leader: 300, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
W20240501 20:09:58.223555 20408 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40850: Illegal state: replica 28b7e229ab3d4f97a09ca51ac037b008 is not leader of this config: current role FOLLOWER
I20240501 20:09:58.223766 20451 raft_consensus.cc:3037] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:58.227056 20451 raft_consensus.cc:2454] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 2.
I20240501 20:09:58.329048 20515 leader_election.cc:302] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, 28b7e229ab3d4f97a09ca51ac037b008; no voters:
I20240501 20:09:58.329504 22571 raft_consensus.cc:2781] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:58.329681 22571 raft_consensus.cc:686] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 2 LEADER]: Becoming Leader. State: Replica: 2305b26386ac446da2436463b536d296, State: Running, Role: LEADER
I20240501 20:09:58.329999 22571 consensus_queue.cc:237] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 1.300, Last appended by leader: 300, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:09:58.340133 20174 catalog_manager.cc:5484] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 reported cstate change: term changed from 1 to 2, leader changed from 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) to 2305b26386ac446da2436463b536d296 (127.14.106.195). New cstate: current_term: 2 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:09:58.340854 22019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a6957734c44303929840f208f5d1d3" candidate_uuid: "2305b26386ac446da2436463b536d296" candidate_term: 2 candidate_status { last_received { term: 1 index: 300 } } ignore_live_leader: true dest_uuid: "00e1df10b53f4936b747b168af137e4b"
I20240501 20:09:58.341197 22019 raft_consensus.cc:3037] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:58.356976 22019 raft_consensus.cc:2454] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2305b26386ac446da2436463b536d296 in term 2.
I20240501 20:09:58.409631 22019 raft_consensus.cc:1261] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 2 index: 302. (index mismatch)
I20240501 20:09:58.410446 22966 consensus_queue.cc:1035] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20240501 20:09:58.416399 20450 raft_consensus.cc:1261] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 2 index: 302. (index mismatch)
I20240501 20:09:58.418115 22966 consensus_queue.cc:1035] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20240501 20:09:58.428552 22916 mvcc.cc:205] Tried to move back new op lower bound from 7022977836676554752 to 7022977836362743808. Current Snapshot: MvccSnapshot[applied={T|T < 7022977836676554752}]
I20240501 20:09:58.435981 22925 mvcc.cc:205] Tried to move back new op lower bound from 7022977836676554752 to 7022977836362743808. Current Snapshot: MvccSnapshot[applied={T|T < 7022977836676554752 or (T in {7022977836676554752})}]
I20240501 20:09:59.017141 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:59.087252 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:59.183554 20560 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:59.294858 21740 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:59.303041 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:59.316999 20431 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:59.329339 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:00.146750 20578 consensus_queue.cc:237] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 2.332, Last appended by leader: 300, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:10:00.148501 22022 raft_consensus.cc:1261] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 2 index: 332. Preceding OpId from leader: term: 2 index: 333. (index mismatch)
I20240501 20:10:00.149648 22571 consensus_queue.cc:1035] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20240501 20:10:00.152254 22571 raft_consensus.cc:2932] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 [term 2 LEADER]: Committing config change with OpId 2.333: config changed from index 158 to 333, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:00.158154 22020 raft_consensus.cc:2932] T 62a6957734c44303929840f208f5d1d3 P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Committing config change with OpId 2.333: config changed from index 158 to 333, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:00.159415 20172 catalog_manager.cc:5484] T 62a6957734c44303929840f208f5d1d3 P 2305b26386ac446da2436463b536d296 reported cstate change: config changed from index 158 to 333, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:00.285064 20431 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a6957734c44303929840f208f5d1d3 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7e229ab3d4f97a09ca51ac037b008 not found in new config with opid_index 333) from {username='slave'} at 127.0.0.1:52596
I20240501 20:10:00.297003 23151 tablet_replica.cc:330] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: stopping tablet replica
I20240501 20:10:00.297678 23151 raft_consensus.cc:2229] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.298094 23151 raft_consensus.cc:2258] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.300854 23151 ts_tablet_manager.cc:1875] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:00.313201 23151 ts_tablet_manager.cc:1888] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.332
I20240501 20:10:00.314458 23151 log.cc:1196] T 62a6957734c44303929840f208f5d1d3 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/62a6957734c44303929840f208f5d1d3
I20240501 20:10:00.315735 20164 catalog_manager.cc:4830] TS 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): tablet 62a6957734c44303929840f208f5d1d3 (table rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]) successfully deleted
I20240501 20:10:01.497493 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:01.505470 21740 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:01.634955 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:01.648609 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:01.656262 20431 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:10:01.673091 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:01.691722 20560 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:02.532755 20450 consensus_queue.cc:237] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 472, Committed index: 472, Last appended: 1.472, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:10:02.543998 22021 raft_consensus.cc:1261] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 472. Preceding OpId from leader: term: 1 index: 473. (index mismatch)
I20240501 20:10:02.545301 22100 consensus_queue.cc:1035] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I20240501 20:10:02.548266 22100 raft_consensus.cc:2932] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.473: config changed from index 123 to 473, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:02.553647 22021 raft_consensus.cc:2932] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Committing config change with OpId 1.473: config changed from index 123 to 473, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:02.554438 20173 catalog_manager.cc:5484] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index 123 to 473, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:02.595355 20298 tablet_service.cc:1514] Processing DeleteTablet for tablet f9480642e71d46679005d4f55d2e88f8 with delete_type TABLET_DATA_TOMBSTONED (TS 4f8d9458da0b406db99095564f9cf2ed not found in new config with opid_index 473) from {username='slave'} at 127.0.0.1:34004
I20240501 20:10:02.608605 23254 tablet_replica.cc:330] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: stopping tablet replica
I20240501 20:10:02.613526 23254 raft_consensus.cc:2229] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.613957 23254 raft_consensus.cc:2258] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.615141 23254 ts_tablet_manager.cc:1875] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:02.633353 23254 ts_tablet_manager.cc:1888] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.472
I20240501 20:10:02.636312 23254 log.cc:1196] T f9480642e71d46679005d4f55d2e88f8 P 4f8d9458da0b406db99095564f9cf2ed: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9480642e71d46679005d4f55d2e88f8
I20240501 20:10:02.639500 20163 catalog_manager.cc:4830] TS 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237): tablet f9480642e71d46679005d4f55d2e88f8 (table rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]) successfully deleted
I20240501 20:10:03.868463 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:03.893267 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:03.939216 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:03.988705 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:04.018767 20560 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:04.036700 21740 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:04.095793 20431 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:10:04.401880 20450 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "afabe33672c647059a9c125c76906fac"
dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38412
I20240501 20:10:04.402107 20450 raft_consensus.cc:595] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:04.402582 22100 raft_consensus.cc:979] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: : Instructing follower a7396c6be769436abac350f7d6e86d55 to start an election
I20240501 20:10:04.402715 22100 raft_consensus.cc:1067] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Signalling peer a7396c6be769436abac350f7d6e86d55 to start an election
I20240501 20:10:04.411585 21759 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "afabe33672c647059a9c125c76906fac"
dest_uuid: "a7396c6be769436abac350f7d6e86d55"
from {username='slave'} at 127.14.106.194:48153
I20240501 20:10:04.411947 21759 raft_consensus.cc:482] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:04.412124 21759 raft_consensus.cc:3037] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:04.415588 21759 raft_consensus.cc:504] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
W20240501 20:10:04.416538 20514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 89822.4us
I20240501 20:10:04.500908 21759 leader_election.cc:288] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 2 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237), 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:10:04.501307 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afabe33672c647059a9c125c76906fac" candidate_uuid: "a7396c6be769436abac350f7d6e86d55" candidate_term: 2 candidate_status { last_received { term: 1 index: 422 } } ignore_live_leader: true dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
I20240501 20:10:04.501368 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afabe33672c647059a9c125c76906fac" candidate_uuid: "a7396c6be769436abac350f7d6e86d55" candidate_term: 2 candidate_status { last_received { term: 1 index: 422 } } ignore_live_leader: true dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:10:04.501562 20451 raft_consensus.cc:3032] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:04.501622 20451 raft_consensus.cc:729] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:10:04.501844 20320 raft_consensus.cc:3037] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:04.501883 20451 consensus_queue.cc:260] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 422, Committed index: 422, Last appended: 1.422, Last appended by leader: 422, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:04.502182 20451 raft_consensus.cc:3037] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:04.505100 20320 raft_consensus.cc:2454] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7396c6be769436abac350f7d6e86d55 in term 2.
I20240501 20:10:04.505985 20451 raft_consensus.cc:2454] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7396c6be769436abac350f7d6e86d55 in term 2.
I20240501 20:10:04.509320 21693 leader_election.cc:302] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, a7396c6be769436abac350f7d6e86d55; no voters:
I20240501 20:10:04.512560 22673 raft_consensus.cc:2781] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:04.524304 22673 raft_consensus.cc:686] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Becoming Leader. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Running, Role: LEADER
I20240501 20:10:04.524777 22673 consensus_queue.cc:237] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 422, Committed index: 422, Last appended: 1.422, Last appended by leader: 422, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:04.529531 20173 catalog_manager.cc:5484] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 reported cstate change: term changed from 1 to 2, leader changed from 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) to a7396c6be769436abac350f7d6e86d55 (127.14.106.197). New cstate: current_term: 2 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
W20240501 20:10:04.537338 20408 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40850: Illegal state: replica 28b7e229ab3d4f97a09ca51ac037b008 is not leader of this config: current role FOLLOWER
W20240501 20:10:04.777807 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58450: Illegal state: replica 4f8d9458da0b406db99095564f9cf2ed is not leader of this config: current role FOLLOWER
I20240501 20:10:04.880945 20318 raft_consensus.cc:1261] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 1 index: 422. Preceding OpId from leader: term: 2 index: 424. (index mismatch)
I20240501 20:10:04.882272 20450 raft_consensus.cc:1261] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 1 index: 422. Preceding OpId from leader: term: 2 index: 424. (index mismatch)
I20240501 20:10:04.882297 22673 consensus_queue.cc:1035] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 423, Last known committed idx: 422, Time since last communication: 0.000s
I20240501 20:10:04.883322 22673 consensus_queue.cc:1035] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 423, Last known committed idx: 422, Time since last communication: 0.000s
I20240501 20:10:05.356456 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:05.430250 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:05.527794 20560 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:05.646310 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:05.701190 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:05.769177 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:05.790416 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:10:06.185997 20384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 47868.2us
W20240501 20:10:06.664131 20384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:36237, user_credentials={real_user=slave}} blocked reactor thread for 122351us
W20240501 20:10:07.182319 20385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:39669, user_credentials={real_user=slave}} blocked reactor thread for 55625.5us
I20240501 20:10:07.320765 21759 consensus_queue.cc:237] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 469, Committed index: 469, Last appended: 2.470, Last appended by leader: 422, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:07.325793 20318 raft_consensus.cc:1261] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 2 index: 470. Preceding OpId from leader: term: 2 index: 471. (index mismatch)
I20240501 20:10:07.326752 23356 consensus_queue.cc:1035] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 471, Last known committed idx: 469, Time since last communication: 0.000s
I20240501 20:10:07.334499 23356 raft_consensus.cc:2932] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Committing config change with OpId 2.471: config changed from index 77 to 471, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:07.338479 20319 raft_consensus.cc:2932] T afabe33672c647059a9c125c76906fac P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Committing config change with OpId 2.471: config changed from index 77 to 471, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:07.347815 20173 catalog_manager.cc:5484] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 reported cstate change: config changed from index 77 to 471, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:07.586339 20431 tablet_service.cc:1514] Processing DeleteTablet for tablet afabe33672c647059a9c125c76906fac with delete_type TABLET_DATA_TOMBSTONED (TS 28b7e229ab3d4f97a09ca51ac037b008 not found in new config with opid_index 471) from {username='slave'} at 127.0.0.1:52596
I20240501 20:10:07.586995 23443 tablet_replica.cc:330] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: stopping tablet replica
I20240501 20:10:07.587240 23443 raft_consensus.cc:2229] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:07.587595 23443 pending_rounds.cc:62] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Trying to abort 1 pending ops.
I20240501 20:10:07.587785 23443 pending_rounds.cc:68] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Aborting op as it isn't in flight: id { term: 2 index: 470 } timestamp: 7022977873157722112 op_type: WRITE_OP write_request { tablet_id: "afabe33672c647059a9c125c76906fac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\010\\\260\023\363r\'\357\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""74f79e22595dcab1" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977873090433024 authz_token { token_data: "\010\257\305\312\261\006\"3\n\005slave\022*\n 194990776984479b928c219a4b993f60\020\001\030\001 \001(\001" signature: "<redacted>""1\224\'\3154\346PL\277\224V\'h\rjr\300\273p\266\3045\226=\261f\007z\231\352$\003\364S\301\337\017\365\002\340o\245\270rK_4\201\253\3245\236\210\302X\356Fs\310\211\"\000\352\030" signing_key_seq_num: 0 } } request_id { client_id: "0074a1ada2fd40d89c01d551dbe0c438" seq_no: 1399 first_incomplete_seq_no: 1399 attempt_no: 0 }
I20240501 20:10:07.588579 23443 raft_consensus.cc:2258] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:07.590101 23443 ts_tablet_manager.cc:1875] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:07.596462 23443 ts_tablet_manager.cc:1888] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.470
I20240501 20:10:07.599678 23443 log.cc:1196] T afabe33672c647059a9c125c76906fac P 28b7e229ab3d4f97a09ca51ac037b008: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/afabe33672c647059a9c125c76906fac
I20240501 20:10:07.604467 20164 catalog_manager.cc:4830] TS 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): tablet afabe33672c647059a9c125c76906fac (table rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]) successfully deleted
I20240501 20:10:09.508174 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:09.542558 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:09.702890 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:09.705075 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:09.720631 20560 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:09.735127 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:09.849750 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:10.518267 20577 consensus_queue.cc:237] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 457, Committed index: 457, Last appended: 1.457, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 458 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:10:10.519923 21625 raft_consensus.cc:1261] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 457. Preceding OpId from leader: term: 1 index: 458. (index mismatch)
I20240501 20:10:10.521659 23322 consensus_queue.cc:1035] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 458, Last known committed idx: 457, Time since last communication: 0.000s
I20240501 20:10:10.525233 23322 raft_consensus.cc:2932] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Committing config change with OpId 1.458: config changed from index 53 to 458, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 458 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:10:10.528479 21625 raft_consensus.cc:2932] T e74fb1e0e017408f946a8df8ae304f13 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Committing config change with OpId 1.458: config changed from index 53 to 458, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 458 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:10:10.530809 20172 catalog_manager.cc:5484] T e74fb1e0e017408f946a8df8ae304f13 P 2305b26386ac446da2436463b536d296 reported cstate change: config changed from index 53 to 458, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 458 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:10:10.611966 20431 tablet_service.cc:1514] Processing DeleteTablet for tablet e74fb1e0e017408f946a8df8ae304f13 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7e229ab3d4f97a09ca51ac037b008 not found in new config with opid_index 458) from {username='slave'} at 127.0.0.1:52596
I20240501 20:10:10.632211 23540 tablet_replica.cc:330] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: stopping tablet replica
I20240501 20:10:10.632529 23540 raft_consensus.cc:2229] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:10.632750 23540 raft_consensus.cc:2258] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:10.633821 23540 ts_tablet_manager.cc:1875] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:10.643632 23540 ts_tablet_manager.cc:1888] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.457
I20240501 20:10:10.643834 23540 log.cc:1196] T e74fb1e0e017408f946a8df8ae304f13 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/e74fb1e0e017408f946a8df8ae304f13
I20240501 20:10:10.644932 20164 catalog_manager.cc:4830] TS 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): tablet e74fb1e0e017408f946a8df8ae304f13 (table rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]) successfully deleted
I20240501 20:10:11.749948 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:11.917064 22149 log.cc:924] Time spent T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: Append to log took a long time: real 0.178s user 0.014s sys 0.000s
I20240501 20:10:11.932727 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:11.941491 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:11.951927 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:11.952883 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:10:11.953766 20384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:36237, user_credentials={real_user=slave}} blocked reactor thread for 109687us
I20240501 20:10:11.954947 20560 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:11.956694 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:12.638988 20451 consensus_queue.cc:237] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 488, Committed index: 488, Last appended: 1.489, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:10:12.641203 22020 raft_consensus.cc:1261] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 489. Preceding OpId from leader: term: 1 index: 490. (index mismatch)
I20240501 20:10:12.641927 22100 consensus_queue.cc:1035] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 490, Last known committed idx: 488, Time since last communication: 0.000s
I20240501 20:10:12.644817 23432 raft_consensus.cc:2932] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.490: config changed from index 61 to 490, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:12.651540 20172 catalog_manager.cc:5484] T ee446411ba4e4da2b295080da28d40b9 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index 61 to 490, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:12.654549 22020 raft_consensus.cc:2932] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Committing config change with OpId 1.490: config changed from index 61 to 490, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:12.741632 20298 tablet_service.cc:1514] Processing DeleteTablet for tablet ee446411ba4e4da2b295080da28d40b9 with delete_type TABLET_DATA_TOMBSTONED (TS 4f8d9458da0b406db99095564f9cf2ed not found in new config with opid_index 490) from {username='slave'} at 127.0.0.1:34004
I20240501 20:10:12.742388 23634 tablet_replica.cc:330] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: stopping tablet replica
I20240501 20:10:12.742655 23634 raft_consensus.cc:2229] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:12.774185 23634 pending_rounds.cc:62] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: Trying to abort 1 pending ops.
I20240501 20:10:12.774437 23634 pending_rounds.cc:68] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: Aborting op as it isn't in flight: id { term: 1 index: 489 } timestamp: 7022977894956818432 op_type: WRITE_OP write_request { tablet_id: "ee446411ba4e4da2b295080da28d40b9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000L*\216b\346\301\351%\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""911f2904297b9cc8" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977894917881856 authz_token { token_data: "\010\257\305\312\261\006\"3\n\005slave\022*\n 9e0813398d56456bb972f8bd0cebde2c\020\001\030\001 \001(\001" signature: "<redacted>""(G\220u8j\373\255`\003\\y\016x\252?\270K*\214\247\334U\t\223S\270[e\223\257\007e\352E\337B\3550x7.\254\304\321x\325\023w\370\\_.3;A\256\340\270\255!,\263\301" signing_key_seq_num: 0 } } request_id { client_id: "ed55082716704d278e29fa019b04f1ce" seq_no: 1444 first_incomplete_seq_no: 1444 attempt_no: 0 }
I20240501 20:10:12.775344 23634 raft_consensus.cc:2258] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:12.776775 23634 ts_tablet_manager.cc:1875] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:12.828405 23634 ts_tablet_manager.cc:1888] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.489
I20240501 20:10:12.828637 23634 log.cc:1196] T ee446411ba4e4da2b295080da28d40b9 P 4f8d9458da0b406db99095564f9cf2ed: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/ee446411ba4e4da2b295080da28d40b9
I20240501 20:10:12.829993 20163 catalog_manager.cc:4830] TS 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237): tablet ee446411ba4e4da2b295080da28d40b9 (table rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]) successfully deleted
W20240501 20:10:13.495817 20517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:36027, user_credentials={real_user=slave}} blocked reactor thread for 56789us
W20240501 20:10:13.594177 20256 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:36027, user_credentials={real_user=slave}} blocked reactor thread for 185316us
I20240501 20:10:13.943280 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:13.968819 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:13.975603 20560 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:14.073434 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:14.105495 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:14.271180 20254 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35301, user_credentials={real_user=slave}} blocked reactor thread for 73163.8us
I20240501 20:10:14.272228 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:14.340283 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:14.662360 20578 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed"
dest_uuid: "2305b26386ac446da2436463b536d296"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39600
I20240501 20:10:14.662601 20578 raft_consensus.cc:595] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Received request to transfer leadership
W20240501 20:10:14.747125 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.756448 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.764860 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.784947 20538 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.796239 20538 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.808713 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.821489 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.850755 20538 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.865933 20538 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.881801 20538 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.897492 20538 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.915287 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.929466 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.956463 20538 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:14.978973 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:15.025919 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:15.049594 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:15.068315 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
W20240501 20:10:15.092680 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
I20240501 20:10:15.098649 23653 raft_consensus.cc:979] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: : Instructing follower a7396c6be769436abac350f7d6e86d55 to start an election
I20240501 20:10:15.098896 23653 raft_consensus.cc:1067] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Signalling peer a7396c6be769436abac350f7d6e86d55 to start an election
I20240501 20:10:15.100636 21759 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed"
dest_uuid: "a7396c6be769436abac350f7d6e86d55"
from {username='slave'} at 127.14.106.195:40117
I20240501 20:10:15.100865 21759 raft_consensus.cc:482] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:15.101004 21759 raft_consensus.cc:3037] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.103725 21759 raft_consensus.cc:504] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
W20240501 20:10:15.117264 20539 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Service unavailable: leader transfer in progress
I20240501 20:10:15.118074 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed" candidate_uuid: "a7396c6be769436abac350f7d6e86d55" candidate_term: 2 candidate_status { last_received { term: 1 index: 495 } } ignore_live_leader: true dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:10:15.118252 20319 raft_consensus.cc:3037] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.118731 21759 leader_election.cc:288] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 2 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237), 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:10:15.123682 20319 raft_consensus.cc:2454] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7396c6be769436abac350f7d6e86d55 in term 2.
I20240501 20:10:15.124415 21692 leader_election.cc:302] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed, a7396c6be769436abac350f7d6e86d55; no voters:
I20240501 20:10:15.124904 23638 raft_consensus.cc:2781] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:15.124857 20578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed" candidate_uuid: "a7396c6be769436abac350f7d6e86d55" candidate_term: 2 candidate_status { last_received { term: 1 index: 495 } } ignore_live_leader: true dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:10:15.125033 23638 raft_consensus.cc:686] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Becoming Leader. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Running, Role: LEADER
I20240501 20:10:15.125056 20578 raft_consensus.cc:3032] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:15.125146 20578 raft_consensus.cc:729] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Running, Role: LEADER
I20240501 20:10:15.125284 23638 consensus_queue.cc:237] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 495, Committed index: 495, Last appended: 1.495, Last appended by leader: 495, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:15.125401 20578 consensus_queue.cc:260] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 495, Committed index: 495, Last appended: 1.495, Last appended by leader: 495, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:15.125756 20578 raft_consensus.cc:3037] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.127658 20174 catalog_manager.cc:5484] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 reported cstate change: term changed from 1 to 2, leader changed from 2305b26386ac446da2436463b536d296 (127.14.106.195) to a7396c6be769436abac350f7d6e86d55 (127.14.106.197). New cstate: current_term: 2 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:15.133723 20578 raft_consensus.cc:2454] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7396c6be769436abac350f7d6e86d55 in term 2.
W20240501 20:10:15.143107 20540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52102: Illegal state: replica 2305b26386ac446da2436463b536d296 is not leader of this config: current role FOLLOWER
W20240501 20:10:15.231691 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48460: Illegal state: replica 4f8d9458da0b406db99095564f9cf2ed is not leader of this config: current role FOLLOWER
I20240501 20:10:15.310724 20581 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 1 index: 495. Preceding OpId from leader: term: 2 index: 497. (index mismatch)
I20240501 20:10:15.310745 20318 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 1 index: 495. Preceding OpId from leader: term: 2 index: 497. (index mismatch)
I20240501 20:10:15.314266 23566 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240501 20:10:15.315795 23638 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240501 20:10:15.534106 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:15.579408 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:15.726739 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:15.758584 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:15.771533 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:15.782891 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:15.819980 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:15.889096 20385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:39669, user_credentials={real_user=slave}} blocked reactor thread for 72863us
I20240501 20:10:16.459507 21759 consensus_queue.cc:237] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 509, Committed index: 509, Last appended: 2.509, Last appended by leader: 495, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 510 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:16.461934 20319 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 2 index: 509. Preceding OpId from leader: term: 2 index: 510. (index mismatch)
I20240501 20:10:16.462769 23566 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 510, Last known committed idx: 509, Time since last communication: 0.000s
I20240501 20:10:16.465806 23566 raft_consensus.cc:2932] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Committing config change with OpId 2.510: config changed from index 98 to 510, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 510 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:16.474483 20319 raft_consensus.cc:2932] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Committing config change with OpId 2.510: config changed from index 98 to 510, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 510 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:16.476796 20173 catalog_manager.cc:5484] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 reported cstate change: config changed from index 98 to 510, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New cstate: current_term: 2 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: 510 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:16.632607 20560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a45309550b449c1bf0ba233c24140ed with delete_type TABLET_DATA_TOMBSTONED (TS 2305b26386ac446da2436463b536d296 not found in new config with opid_index 510) from {username='slave'} at 127.0.0.1:49878
I20240501 20:10:16.656877 23804 tablet_replica.cc:330] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: stopping tablet replica
I20240501 20:10:16.666457 23804 raft_consensus.cc:2229] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:16.669446 23804 raft_consensus.cc:2258] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:16.671947 23804 ts_tablet_manager.cc:1875] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:16.712006 23804 ts_tablet_manager.cc:1888] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.509
I20240501 20:10:16.712136 23804 log.cc:1196] T 5a45309550b449c1bf0ba233c24140ed P 2305b26386ac446da2436463b536d296: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/5a45309550b449c1bf0ba233c24140ed
I20240501 20:10:16.722929 20164 catalog_manager.cc:4830] TS 2305b26386ac446da2436463b536d296 (127.14.106.195:35301): tablet 5a45309550b449c1bf0ba233c24140ed (table rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]) successfully deleted
I20240501 20:10:17.144942 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:17.177035 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:17.283300 20385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:39669, user_credentials={real_user=slave}} blocked reactor thread for 51100.2us
I20240501 20:10:17.381060 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:17.409088 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:17.412986 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:17.433238 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:17.494050 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:17.866029 20451 consensus_queue.cc:237] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 558, Committed index: 558, Last appended: 1.558, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 559 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:17.869243 21891 raft_consensus.cc:1261] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 558. Preceding OpId from leader: term: 1 index: 559. (index mismatch)
I20240501 20:10:17.870700 23363 consensus_queue.cc:1035] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 559, Last known committed idx: 558, Time since last communication: 0.000s
I20240501 20:10:17.876294 23286 raft_consensus.cc:2932] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.559: config changed from index 58 to 559, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 559 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:17.877074 21891 raft_consensus.cc:2932] T 2e414a3678e646c6b5c6c0dad4fac30e P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Committing config change with OpId 1.559: config changed from index 58 to 559, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 559 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:17.884804 20173 catalog_manager.cc:5484] T 2e414a3678e646c6b5c6c0dad4fac30e P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index 58 to 559, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 559 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:17.931771 20560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e414a3678e646c6b5c6c0dad4fac30e with delete_type TABLET_DATA_TOMBSTONED (TS 2305b26386ac446da2436463b536d296 not found in new config with opid_index 559) from {username='slave'} at 127.0.0.1:49878
I20240501 20:10:17.956988 23883 tablet_replica.cc:330] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: stopping tablet replica
I20240501 20:10:17.958739 23883 raft_consensus.cc:2229] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:17.959022 23883 raft_consensus.cc:2258] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:17.960458 23883 ts_tablet_manager.cc:1875] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:17.994750 23883 ts_tablet_manager.cc:1888] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.558
I20240501 20:10:17.995008 23883 log.cc:1196] T 2e414a3678e646c6b5c6c0dad4fac30e P 2305b26386ac446da2436463b536d296: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/2e414a3678e646c6b5c6c0dad4fac30e
I20240501 20:10:17.997743 20164 catalog_manager.cc:4830] TS 2305b26386ac446da2436463b536d296 (127.14.106.195:35301): tablet 2e414a3678e646c6b5c6c0dad4fac30e (table rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]) successfully deleted
I20240501 20:10:18.962172 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:18.963788 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:19.026731 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:19.125957 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:19.138430 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:19.283427 20514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 199017us
I20240501 20:10:19.295307 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:19.357059 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:20.245294 20451 consensus_queue.cc:237] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 642, Committed index: 642, Last appended: 1.642, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 643 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:10:20.252322 21626 raft_consensus.cc:1261] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 642. Preceding OpId from leader: term: 1 index: 643. (index mismatch)
I20240501 20:10:20.253324 23501 consensus_queue.cc:1035] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 643, Last known committed idx: 642, Time since last communication: 0.000s
I20240501 20:10:20.258569 23501 raft_consensus.cc:2932] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.643: config changed from index 79 to 643, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 643 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:10:20.259160 21627 raft_consensus.cc:2932] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Committing config change with OpId 1.643: config changed from index 79 to 643, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 643 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:10:20.273660 20174 catalog_manager.cc:5484] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index 79 to 643, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 643 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } }
I20240501 20:10:20.543476 20560 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c119cb173942819ec01f7f3797c9d7 with delete_type TABLET_DATA_TOMBSTONED (TS 2305b26386ac446da2436463b536d296 not found in new config with opid_index 643) from {username='slave'} at 127.0.0.1:49878
I20240501 20:10:20.696408 23991 tablet_replica.cc:330] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: stopping tablet replica
I20240501 20:10:20.696985 23991 raft_consensus.cc:2229] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:20.726892 23991 raft_consensus.cc:2258] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:20.737185 23991 ts_tablet_manager.cc:1875] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:20.777779 23991 ts_tablet_manager.cc:1888] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.642
I20240501 20:10:20.778056 23991 log.cc:1196] T e4c119cb173942819ec01f7f3797c9d7 P 2305b26386ac446da2436463b536d296: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/e4c119cb173942819ec01f7f3797c9d7
I20240501 20:10:20.787573 20164 catalog_manager.cc:4830] TS 2305b26386ac446da2436463b536d296 (127.14.106.195:35301): tablet e4c119cb173942819ec01f7f3797c9d7 (table rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]) successfully deleted
W20240501 20:10:21.932729 20385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35301, user_credentials={real_user=slave}} blocked reactor thread for 43828.6us
I20240501 20:10:21.934895 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:21.942549 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:21.995494 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:22.114324 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:22.131120 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:22.133301 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:22.152932 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:22.663647 21759 consensus_queue.cc:237] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:10:22.672699 22021 raft_consensus.cc:1261] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:22.673447 23950 consensus_queue.cc:1035] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.694103 24002 raft_consensus.cc:2932] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:22.698840 22021 raft_consensus.cc:2932] T e4dff67082a6473f994c09821a0d7a3d P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:22.704619 20174 catalog_manager.cc:5484] T e4dff67082a6473f994c09821a0d7a3d P a7396c6be769436abac350f7d6e86d55 reported cstate change: config changed from index 3 to 4, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New cstate: current_term: 2 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } }
I20240501 20:10:22.788987 20298 tablet_service.cc:1514] Processing DeleteTablet for tablet e4dff67082a6473f994c09821a0d7a3d with delete_type TABLET_DATA_TOMBSTONED (TS 4f8d9458da0b406db99095564f9cf2ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34004
I20240501 20:10:22.791114 24093 tablet_replica.cc:330] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: stopping tablet replica
I20240501 20:10:22.791301 24093 raft_consensus.cc:2229] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:22.791484 24093 raft_consensus.cc:2258] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:22.792047 24093 ts_tablet_manager.cc:1875] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:22.814342 24093 ts_tablet_manager.cc:1888] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 20:10:22.814569 24093 log.cc:1196] T e4dff67082a6473f994c09821a0d7a3d P 4f8d9458da0b406db99095564f9cf2ed: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4dff67082a6473f994c09821a0d7a3d
I20240501 20:10:22.815430 20163 catalog_manager.cc:4830] TS 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237): tablet e4dff67082a6473f994c09821a0d7a3d (table TestTable [id=905e41f6a17144258cf5ffc0d8e5e6ad]) successfully deleted
I20240501 20:10:23.519438 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:23.646616 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:23.673599 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:23.692471 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:23.702116 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:23.759215 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:23.858156 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:10:23.948652 20515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:39669, user_credentials={real_user=slave}} blocked reactor thread for 64861us
I20240501 20:10:24.259850 20450 consensus_queue.cc:237] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 810, Committed index: 810, Last appended: 1.810, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:24.261761 21891 raft_consensus.cc:1261] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 810. Preceding OpId from leader: term: 1 index: 811. (index mismatch)
I20240501 20:10:24.262531 23501 consensus_queue.cc:1035] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 811, Last known committed idx: 810, Time since last communication: 0.000s
I20240501 20:10:24.265164 24035 raft_consensus.cc:2932] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.811: config changed from index 159 to 811, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:24.265892 21892 raft_consensus.cc:2932] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Committing config change with OpId 1.811: config changed from index 159 to 811, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:24.270325 20173 catalog_manager.cc:5484] T 01bf239579d84c07adff90ca8f87824d P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index 159 to 811, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:24.373714 20560 tablet_service.cc:1514] Processing DeleteTablet for tablet 01bf239579d84c07adff90ca8f87824d with delete_type TABLET_DATA_TOMBSTONED (TS 2305b26386ac446da2436463b536d296 not found in new config with opid_index 811) from {username='slave'} at 127.0.0.1:49878
I20240501 20:10:24.417445 24181 tablet_replica.cc:330] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: stopping tablet replica
I20240501 20:10:24.417759 24181 raft_consensus.cc:2229] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:24.417981 24181 raft_consensus.cc:2258] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.420258 24181 ts_tablet_manager.cc:1875] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:24.447333 24181 ts_tablet_manager.cc:1888] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.810
I20240501 20:10:24.450270 24181 log.cc:1196] T 01bf239579d84c07adff90ca8f87824d P 2305b26386ac446da2436463b536d296: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/01bf239579d84c07adff90ca8f87824d
I20240501 20:10:24.452234 20164 catalog_manager.cc:4830] TS 2305b26386ac446da2436463b536d296 (127.14.106.195:35301): tablet 01bf239579d84c07adff90ca8f87824d (table rebalance_test_table_0 [id=194990776984479b928c219a4b993f60]) successfully deleted
I20240501 20:10:25.192754 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:25.207088 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:25.234187 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:25.261289 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:25.324542 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:25.409763 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:25.557276 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:26.047279 20581 consensus_queue.cc:237] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 747, Committed index: 747, Last appended: 1.747, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:26.049144 21889 raft_consensus.cc:1261] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 747. Preceding OpId from leader: term: 1 index: 748. (index mismatch)
I20240501 20:10:26.049753 24151 consensus_queue.cc:1035] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 748, Last known committed idx: 747, Time since last communication: 0.000s
I20240501 20:10:26.052248 23720 raft_consensus.cc:2932] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Committing config change with OpId 1.748: config changed from index 88 to 748, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:26.053153 21889 raft_consensus.cc:2932] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Committing config change with OpId 1.748: config changed from index 88 to 748, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:26.060554 20174 catalog_manager.cc:5484] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 reported cstate change: config changed from index 88 to 748, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
W20240501 20:10:26.114724 20387 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:36027, user_credentials={real_user=slave}} blocked reactor thread for 73127us
W20240501 20:10:26.115548 20384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 73664.3us
W20240501 20:10:26.165134 20387 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:36027, user_credentials={real_user=slave}} blocked reactor thread for 45023.1us
I20240501 20:10:26.167368 20431 tablet_service.cc:1514] Processing DeleteTablet for tablet e8477d3f66b940538bb74f2cf87a3d47 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7e229ab3d4f97a09ca51ac037b008 not found in new config with opid_index 748) from {username='slave'} at 127.0.0.1:52596
I20240501 20:10:26.182278 24275 tablet_replica.cc:330] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: stopping tablet replica
I20240501 20:10:26.182515 24275 raft_consensus.cc:2229] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.182754 24275 raft_consensus.cc:2258] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.188535 24275 ts_tablet_manager.cc:1875] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.197203 24275 ts_tablet_manager.cc:1888] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.747
I20240501 20:10:26.197463 24275 log.cc:1196] T e8477d3f66b940538bb74f2cf87a3d47 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8477d3f66b940538bb74f2cf87a3d47
I20240501 20:10:26.211750 20164 catalog_manager.cc:4830] TS 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): tablet e8477d3f66b940538bb74f2cf87a3d47 (table rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]) successfully deleted
I20240501 20:10:29.196313 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:29.252830 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:29.261037 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:29.276432 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:29.301108 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:29.345777 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:29.348214 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:29.805266 20451 consensus_queue.cc:237] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 838, Committed index: 838, Last appended: 1.838, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
W20240501 20:10:29.809667 20387 consensus_peers.cc:485] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Couldn't send request to peer a7396c6be769436abac350f7d6e86d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4c119cb173942819ec01f7f3797c9d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.817440 21626 raft_consensus.cc:1261] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 838. Preceding OpId from leader: term: 1 index: 839. (index mismatch)
I20240501 20:10:29.818078 24269 consensus_queue.cc:1035] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 839, Last known committed idx: 838, Time since last communication: 0.000s
I20240501 20:10:29.821810 24267 raft_consensus.cc:2932] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.839: config changed from index 643 to 839, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New config: { opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:29.831892 21626 raft_consensus.cc:2932] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Committing config change with OpId 1.839: config changed from index 643 to 839, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New config: { opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:29.838963 20173 catalog_manager.cc:5484] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index 643 to 839, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:30.133464 20320 consensus_queue.cc:237] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 853, Committed index: 853, Last appended: 1.854, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:30.139773 20577 raft_consensus.cc:1261] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9458da0b406db99095564f9cf2ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 854. Preceding OpId from leader: term: 1 index: 855. (index mismatch)
I20240501 20:10:30.140700 24387 consensus_queue.cc:1035] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 855, Last known committed idx: 853, Time since last communication: 0.000s
I20240501 20:10:30.143429 24098 raft_consensus.cc:2932] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Committing config change with OpId 1.855: config changed from index -1 to 855, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New config: { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:30.150902 20174 catalog_manager.cc:5484] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: config changed from index -1 to 855, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "4f8d9458da0b406db99095564f9cf2ed" committed_config { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:30.155098 20577 raft_consensus.cc:2932] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Committing config change with OpId 1.855: config changed from index -1 to 855, VOTER 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198) added. New config: { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
W20240501 20:10:30.192057 20253 consensus_peers.cc:485] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed -> Peer 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198:43715): Couldn't send request to peer 133d5e0cbb234be1a5752a784d6d934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9ccf34e292a480aa7283631edb5a0d2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.351995 24485 ts_tablet_manager.cc:902] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:10:30.443637 20451 consensus_queue.cc:237] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 829, Committed index: 829, Last appended: 1.830, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
W20240501 20:10:30.445986 20387 consensus_peers.cc:485] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Couldn't send request to peer a7396c6be769436abac350f7d6e86d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72363ecf8c08466b8b7f5f4a2d07e1d4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.449731 20319 raft_consensus.cc:1261] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 830. Preceding OpId from leader: term: 1 index: 831. (index mismatch)
I20240501 20:10:30.450286 24265 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 831, Last known committed idx: 829, Time since last communication: 0.000s
I20240501 20:10:30.453567 24485 tablet_copy_client.cc:308] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:10:30.455145 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet e4c119cb173942819ec01f7f3797c9d7 from peer a7396c6be769436abac350f7d6e86d55 ({username='slave'} at 127.14.106.197:43223)
I20240501 20:10:30.455283 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet e4c119cb173942819ec01f7f3797c9d7 from peer a7396c6be769436abac350f7d6e86d55 at {username='slave'} at 127.14.106.197:43223: session id = a7396c6be769436abac350f7d6e86d55-e4c119cb173942819ec01f7f3797c9d7
I20240501 20:10:30.457983 20471 tablet_copy_source_session.cc:205] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.471128 24485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c119cb173942819ec01f7f3797c9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.483305 24265 raft_consensus.cc:2932] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.831: config changed from index -1 to 831, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New config: { opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:30.494174 20320 raft_consensus.cc:2932] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Committing config change with OpId 1.831: config changed from index -1 to 831, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New config: { opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:30.498426 20174 catalog_manager.cc:5484] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 reported cstate change: config changed from index -1 to 831, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:30.532954 24485 tablet_copy_client.cc:783] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.533419 24485 tablet_copy_client.cc:647] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.542148 24485 tablet_copy_client.cc:523] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.546869 24485 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: Bootstrap starting.
I20240501 20:10:30.576157 24503 ts_tablet_manager.cc:902] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: Initiating tablet copy from peer 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:10:30.872225 24503 tablet_copy_client.cc:308] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:36237
I20240501 20:10:30.892228 20340 tablet_copy_service.cc:140] P 4f8d9458da0b406db99095564f9cf2ed: Received BeginTabletCopySession request for tablet b9ccf34e292a480aa7283631edb5a0d2 from peer 133d5e0cbb234be1a5752a784d6d934a ({username='slave'} at 127.14.106.198:48449)
I20240501 20:10:30.892378 20340 tablet_copy_service.cc:161] P 4f8d9458da0b406db99095564f9cf2ed: Beginning new tablet copy session on tablet b9ccf34e292a480aa7283631edb5a0d2 from peer 133d5e0cbb234be1a5752a784d6d934a at {username='slave'} at 127.14.106.198:48449: session id = 133d5e0cbb234be1a5752a784d6d934a-b9ccf34e292a480aa7283631edb5a0d2
I20240501 20:10:30.896510 20340 tablet_copy_source_session.cc:205] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.897938 24503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ccf34e292a480aa7283631edb5a0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.922765 24503 tablet_copy_client.cc:783] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.924178 24503 tablet_copy_client.cc:647] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.936146 24503 tablet_copy_client.cc:523] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.942721 24503 tablet_bootstrap.cc:492] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap starting.
I20240501 20:10:31.065498 24523 ts_tablet_manager.cc:902] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:10:31.118486 24523 tablet_copy_client.cc:308] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:10:31.119416 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 from peer a7396c6be769436abac350f7d6e86d55 ({username='slave'} at 127.14.106.197:43223)
I20240501 20:10:31.119573 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 from peer a7396c6be769436abac350f7d6e86d55 at {username='slave'} at 127.14.106.197:43223: session id = a7396c6be769436abac350f7d6e86d55-72363ecf8c08466b8b7f5f4a2d07e1d4
I20240501 20:10:31.122397 20471 tablet_copy_source_session.cc:205] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:31.124002 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.130165 24523 tablet_copy_client.cc:783] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:31.133008 24523 tablet_copy_client.cc:647] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:31.137126 24523 tablet_copy_client.cc:523] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:31.145761 24523 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: Bootstrap starting.
I20240501 20:10:32.045715 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:32.280193 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:32.284201 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:32.314476 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:32.355410 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:32.394750 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:32.433851 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:32.730894 20514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 48774.5us
W20240501 20:10:32.983007 20387 consensus_peers.cc:485] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Couldn't send request to peer a7396c6be769436abac350f7d6e86d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:33.078802 20253 consensus_peers.cc:485] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed -> Peer 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198:43715): Couldn't send request to peer 133d5e0cbb234be1a5752a784d6d934a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:33.306602 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:33.395473 20254 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35301, user_credentials={real_user=slave}} blocked reactor thread for 235679us
I20240501 20:10:33.426416 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:33.426493 20387 consensus_peers.cc:485] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Couldn't send request to peer a7396c6be769436abac350f7d6e86d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:33.434105 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:33.459693 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:33.495200 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:33.496235 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:33.524112 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:33.845324 24485 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=857 overwritten=0 applied=857 ignored=0} inserts{seen=1486 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:33.845940 24485 tablet_bootstrap.cc:492] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: Bootstrap complete.
I20240501 20:10:33.846369 24485 ts_tablet_manager.cc:1367] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: Time spent bootstrapping tablet: real 3.300s user 1.001s sys 0.064s
I20240501 20:10:33.847311 24485 raft_consensus.cc:348] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:33.847589 24485 raft_consensus.cc:729] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.848340 24485 consensus_queue.cc:260] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 857, Last appended: 1.857, Last appended by leader: 857, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:33.848985 24485 ts_tablet_manager.cc:1398] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:33.849900 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session a7396c6be769436abac350f7d6e86d55-e4c119cb173942819ec01f7f3797c9d7 received from {username='slave'} at 127.14.106.197:43223
I20240501 20:10:33.850021 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session a7396c6be769436abac350f7d6e86d55-e4c119cb173942819ec01f7f3797c9d7 on tablet e4c119cb173942819ec01f7f3797c9d7 with peer a7396c6be769436abac350f7d6e86d55
W20240501 20:10:33.865567 20514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 295672us
W20240501 20:10:33.874424 20517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:36027, user_credentials={real_user=slave}} blocked reactor thread for 303416us
I20240501 20:10:33.980464 24523 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=841 overwritten=0 applied=841 ignored=0} inserts{seen=1513 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:33.981078 24523 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: Bootstrap complete.
I20240501 20:10:33.981935 24523 ts_tablet_manager.cc:1367] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: Time spent bootstrapping tablet: real 2.836s user 1.002s sys 0.077s
I20240501 20:10:33.987751 24523 raft_consensus.cc:348] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:33.988095 24523 raft_consensus.cc:729] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.988406 24523 consensus_queue.cc:260] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 841, Last appended: 1.841, Last appended by leader: 841, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:33.989066 24523 ts_tablet_manager.cc:1398] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 20:10:33.990650 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session a7396c6be769436abac350f7d6e86d55-72363ecf8c08466b8b7f5f4a2d07e1d4 received from {username='slave'} at 127.14.106.197:43223
I20240501 20:10:33.990759 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session a7396c6be769436abac350f7d6e86d55-72363ecf8c08466b8b7f5f4a2d07e1d4 on tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 with peer a7396c6be769436abac350f7d6e86d55
I20240501 20:10:34.012107 21759 raft_consensus.cc:1203] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.838->[1.839-1.892] Dedup: 1.857->[1.858-1.892]
I20240501 20:10:34.390856 21759 raft_consensus.cc:1203] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.830->[1.831-1.878] Dedup: 1.841->[1.842-1.878]
I20240501 20:10:34.598130 24503 tablet_bootstrap.cc:492] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap replayed 1/1 log segments. Stats: ops{read=874 overwritten=0 applied=874 ignored=0} inserts{seen=1505 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:34.602869 24503 tablet_bootstrap.cc:492] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: Bootstrap complete.
I20240501 20:10:34.603430 24503 ts_tablet_manager.cc:1367] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: Time spent bootstrapping tablet: real 3.661s user 1.061s sys 0.076s
I20240501 20:10:34.605650 24503 raft_consensus.cc:348] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:34.619243 24503 raft_consensus.cc:729] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133d5e0cbb234be1a5752a784d6d934a, State: Initialized, Role: FOLLOWER
I20240501 20:10:34.619705 24503 consensus_queue.cc:260] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 874, Last appended: 1.874, Last appended by leader: 874, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:34.627738 24503 ts_tablet_manager.cc:1398] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a: Time spent starting tablet: real 0.023s user 0.001s sys 0.002s
I20240501 20:10:34.629190 20340 tablet_copy_service.cc:342] P 4f8d9458da0b406db99095564f9cf2ed: Request end of tablet copy session 133d5e0cbb234be1a5752a784d6d934a-b9ccf34e292a480aa7283631edb5a0d2 received from {username='slave'} at 127.14.106.198:48449
I20240501 20:10:34.629302 20340 tablet_copy_service.cc:433] P 4f8d9458da0b406db99095564f9cf2ed: ending tablet copy session 133d5e0cbb234be1a5752a784d6d934a-b9ccf34e292a480aa7283631edb5a0d2 on tablet b9ccf34e292a480aa7283631edb5a0d2 with peer 133d5e0cbb234be1a5752a784d6d934a
I20240501 20:10:35.154032 21890 raft_consensus.cc:1203] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.854->[1.855-1.945] Dedup: 1.874->[1.875-1.945]
I20240501 20:10:35.435736 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:35.515273 20431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:35.538486 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:35.548875 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:35.694330 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:35.725507 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:35.898228 20256 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:36027, user_credentials={real_user=slave}} blocked reactor thread for 112519us
I20240501 20:10:35.936053 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:36.249681 20451 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4"
dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43440
I20240501 20:10:36.249897 20451 raft_consensus.cc:595] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:36.251313 24269 raft_consensus.cc:979] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: : Instructing follower 4f8d9458da0b406db99095564f9cf2ed to start an election
I20240501 20:10:36.251446 24269 raft_consensus.cc:1067] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Signalling peer 4f8d9458da0b406db99095564f9cf2ed to start an election
I20240501 20:10:36.258643 20320 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4"
dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
from {username='slave'} at 127.14.106.194:43407
I20240501 20:10:36.266049 20320 raft_consensus.cc:482] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:36.266206 20320 raft_consensus.cc:3037] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:36.269042 20320 raft_consensus.cc:504] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:36.274065 20320 leader_election.cc:288] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 election: Requested vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937), a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
I20240501 20:10:36.299685 21760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4" candidate_uuid: "4f8d9458da0b406db99095564f9cf2ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 925 } } ignore_live_leader: true dest_uuid: "a7396c6be769436abac350f7d6e86d55"
I20240501 20:10:36.300060 21760 raft_consensus.cc:3037] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:36.303690 21760 raft_consensus.cc:2454] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9458da0b406db99095564f9cf2ed in term 2.
I20240501 20:10:36.304351 20256 leader_election.cc:302] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed, a7396c6be769436abac350f7d6e86d55; no voters:
I20240501 20:10:36.304646 24720 raft_consensus.cc:2781] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:36.304780 24720 raft_consensus.cc:686] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 LEADER]: Becoming Leader. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Running, Role: LEADER
I20240501 20:10:36.305032 24720 consensus_queue.cc:237] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 925, Committed index: 925, Last appended: 1.925, Last appended by leader: 925, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:36.308125 20173 catalog_manager.cc:5484] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: term changed from 1 to 2, leader changed from 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) to 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193). New cstate: current_term: 2 leader_uuid: "4f8d9458da0b406db99095564f9cf2ed" committed_config { opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:36.352597 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4" candidate_uuid: "4f8d9458da0b406db99095564f9cf2ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 925 } } ignore_live_leader: true dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
I20240501 20:10:36.352800 20451 raft_consensus.cc:3032] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:36.352864 20451 raft_consensus.cc:729] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:10:36.353081 20451 consensus_queue.cc:260] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 925, Committed index: 925, Last appended: 1.925, Last appended by leader: 925, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:36.353404 20451 raft_consensus.cc:3037] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:36.356704 20451 raft_consensus.cc:2454] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9458da0b406db99095564f9cf2ed in term 2.
W20240501 20:10:36.526072 20410 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40906: Illegal state: replica 28b7e229ab3d4f97a09ca51ac037b008 is not leader of this config: current role FOLLOWER
I20240501 20:10:36.680964 20451 raft_consensus.cc:1261] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Refusing update from remote peer 4f8d9458da0b406db99095564f9cf2ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 925. Preceding OpId from leader: term: 2 index: 927. (index mismatch)
I20240501 20:10:36.684680 24718 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 926, Last known committed idx: 925, Time since last communication: 0.000s
I20240501 20:10:36.704452 21760 raft_consensus.cc:1261] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Refusing update from remote peer 4f8d9458da0b406db99095564f9cf2ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 925. Preceding OpId from leader: term: 2 index: 927. (index mismatch)
I20240501 20:10:36.708822 24718 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 926, Last known committed idx: 925, Time since last communication: 0.000s
W20240501 20:10:37.007040 20514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:43715, user_credentials={real_user=slave}} blocked reactor thread for 51756.5us
I20240501 20:10:37.231703 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:37.263262 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:37.310700 20298 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:37.312623 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:37.314913 20431 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:10:37.443678 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:37.474442 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.261459 20319 consensus_queue.cc:237] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 944, Committed index: 944, Last appended: 2.944, Last appended by leader: 925, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 945 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:38.268486 21759 raft_consensus.cc:1261] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Refusing update from remote peer 4f8d9458da0b406db99095564f9cf2ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 944. Preceding OpId from leader: term: 2 index: 945. (index mismatch)
I20240501 20:10:38.270519 24720 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 945, Last known committed idx: 943, Time since last communication: 0.000s
I20240501 20:10:38.273313 24720 raft_consensus.cc:2932] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 LEADER]: Committing config change with OpId 2.945: config changed from index 831 to 945, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 945 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:38.282513 21760 raft_consensus.cc:2932] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Committing config change with OpId 2.945: config changed from index 831 to 945, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 945 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:38.293706 20174 catalog_manager.cc:5484] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: config changed from index 831 to 945, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "4f8d9458da0b406db99095564f9cf2ed" committed_config { opid_index: 945 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:38.342296 20431 tablet_service.cc:1514] Processing DeleteTablet for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7e229ab3d4f97a09ca51ac037b008 not found in new config with opid_index 945) from {username='slave'} at 127.0.0.1:52596
I20240501 20:10:38.351600 24840 tablet_replica.cc:330] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: stopping tablet replica
I20240501 20:10:38.351940 24840 raft_consensus.cc:2229] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:38.352236 24840 raft_consensus.cc:2258] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:38.353262 24840 ts_tablet_manager.cc:1875] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:38.360020 24840 ts_tablet_manager.cc:1888] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.944
I20240501 20:10:38.360229 24840 log.cc:1196] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/72363ecf8c08466b8b7f5f4a2d07e1d4
I20240501 20:10:38.367784 20164 catalog_manager.cc:4830] TS 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 (table rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]) successfully deleted
I20240501 20:10:39.332456 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.355835 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.504174 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:39.534461 20431 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:10:39.565886 21608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.606360 20298 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:39.616183 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:40.083973 20451 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4c119cb173942819ec01f7f3797c9d7"
dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43478
I20240501 20:10:40.084172 20451 raft_consensus.cc:595] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:40.090867 24265 raft_consensus.cc:979] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: : Instructing follower 79e9c50adfeb44929f5d29524169c6c7 to start an election
I20240501 20:10:40.091023 24265 raft_consensus.cc:1067] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Signalling peer 79e9c50adfeb44929f5d29524169c6c7 to start an election
I20240501 20:10:40.092109 21624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4c119cb173942819ec01f7f3797c9d7"
dest_uuid: "79e9c50adfeb44929f5d29524169c6c7"
from {username='slave'} at 127.14.106.194:58007
I20240501 20:10:40.092373 21624 raft_consensus.cc:482] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:40.092566 21624 raft_consensus.cc:3037] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:40.098325 20410 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40874: Service unavailable: leader transfer in progress
I20240501 20:10:40.100095 21624 raft_consensus.cc:504] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
W20240501 20:10:40.106830 20410 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40874: Service unavailable: leader transfer in progress
W20240501 20:10:40.113086 20410 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40874: Service unavailable: leader transfer in progress
I20240501 20:10:40.117223 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c119cb173942819ec01f7f3797c9d7" candidate_uuid: "79e9c50adfeb44929f5d29524169c6c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1008 } } ignore_live_leader: true dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
I20240501 20:10:40.117403 20451 raft_consensus.cc:3032] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:40.117491 20451 raft_consensus.cc:729] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:10:40.117702 20451 consensus_queue.cc:260] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1008, Committed index: 1008, Last appended: 1.1008, Last appended by leader: 1008, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:40.117931 20451 raft_consensus.cc:3037] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:40.120615 20451 raft_consensus.cc:2454] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e9c50adfeb44929f5d29524169c6c7 in term 2.
I20240501 20:10:40.121173 21562 leader_election.cc:302] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7e229ab3d4f97a09ca51ac037b008, 79e9c50adfeb44929f5d29524169c6c7; no voters:
W20240501 20:10:40.121529 20410 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40874: Illegal state: replica 28b7e229ab3d4f97a09ca51ac037b008 is not leader of this config: current role FOLLOWER
I20240501 20:10:40.122908 24908 raft_consensus.cc:2781] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:40.124872 21624 leader_election.cc:288] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [CANDIDATE]: Term 2 election: Requested vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937), a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
W20240501 20:10:40.139385 20411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40874: Illegal state: replica 28b7e229ab3d4f97a09ca51ac037b008 is not leader of this config: current role FOLLOWER
I20240501 20:10:40.315232 24908 raft_consensus.cc:686] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 2 LEADER]: Becoming Leader. State: Replica: 79e9c50adfeb44929f5d29524169c6c7, State: Running, Role: LEADER
I20240501 20:10:40.316001 24908 consensus_queue.cc:237] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1008, Committed index: 1008, Last appended: 1.1008, Last appended by leader: 1008, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:40.323019 20174 catalog_manager.cc:5484] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 reported cstate change: term changed from 1 to 2, leader changed from 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) to 79e9c50adfeb44929f5d29524169c6c7 (127.14.106.196). New cstate: current_term: 2 leader_uuid: "79e9c50adfeb44929f5d29524169c6c7" committed_config { opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:40.344559 21759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c119cb173942819ec01f7f3797c9d7" candidate_uuid: "79e9c50adfeb44929f5d29524169c6c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1008 } } ignore_live_leader: true dest_uuid: "a7396c6be769436abac350f7d6e86d55"
I20240501 20:10:40.344790 21759 raft_consensus.cc:3037] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:40.347478 21759 raft_consensus.cc:2454] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e9c50adfeb44929f5d29524169c6c7 in term 2.
I20240501 20:10:40.413980 21759 raft_consensus.cc:1261] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Refusing update from remote peer 79e9c50adfeb44929f5d29524169c6c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1007. Preceding OpId from leader: term: 2 index: 1010. (index mismatch)
I20240501 20:10:40.414537 20451 raft_consensus.cc:1261] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Refusing update from remote peer 79e9c50adfeb44929f5d29524169c6c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1008. Preceding OpId from leader: term: 2 index: 1010. (index mismatch)
I20240501 20:10:40.416368 24909 consensus_queue.cc:1035] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1009, Last known committed idx: 1006, Time since last communication: 0.000s
I20240501 20:10:40.417233 24909 consensus_queue.cc:1035] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1009, Last known committed idx: 1008, Time since last communication: 0.000s
I20240501 20:10:40.426940 24752 mvcc.cc:205] Tried to move back new op lower bound from 7022978008727937024 to 7022978008351154176. Current Snapshot: MvccSnapshot[applied={T|T < 7022978008727937024 or (T in {7022978008727937024})}]
I20240501 20:10:40.441205 24831 mvcc.cc:205] Tried to move back new op lower bound from 7022978008727937024 to 7022978008351154176. Current Snapshot: MvccSnapshot[applied={T|T < 7022978008727937024}]
W20240501 20:10:40.583453 21564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:35605, user_credentials={real_user=slave}} blocked reactor thread for 104066us
I20240501 20:10:40.983907 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:41.199959 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:41.223364 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:41.284202 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:41.332747 20298 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:41.455092 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:41.543334 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:10:42.114347 22531 log.cc:924] Time spent T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a: Append to log took a long time: real 0.112s user 0.014s sys 0.000s
I20240501 20:10:42.343677 21626 consensus_queue.cc:237] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1039, Committed index: 1039, Last appended: 2.1039, Last appended by leader: 1008, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1040 OBSOLETE_local: false peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } }
I20240501 20:10:42.347081 21759 raft_consensus.cc:1261] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Refusing update from remote peer 79e9c50adfeb44929f5d29524169c6c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 1039. Preceding OpId from leader: term: 2 index: 1040. (index mismatch)
I20240501 20:10:42.347723 24908 consensus_queue.cc:1035] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1040, Last known committed idx: 1039, Time since last communication: 0.000s
I20240501 20:10:42.351092 24908 raft_consensus.cc:2932] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 [term 2 LEADER]: Committing config change with OpId 2.1040: config changed from index 839 to 1040, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 1040 OBSOLETE_local: false peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:42.359095 20179 catalog_manager.cc:5484] T e4c119cb173942819ec01f7f3797c9d7 P 79e9c50adfeb44929f5d29524169c6c7 reported cstate change: config changed from index 839 to 1040, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "79e9c50adfeb44929f5d29524169c6c7" committed_config { opid_index: 1040 OBSOLETE_local: false peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:42.359474 21759 raft_consensus.cc:2932] T e4c119cb173942819ec01f7f3797c9d7 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Committing config change with OpId 2.1040: config changed from index 839 to 1040, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 1040 OBSOLETE_local: false peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } }
I20240501 20:10:42.498898 20431 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c119cb173942819ec01f7f3797c9d7 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7e229ab3d4f97a09ca51ac037b008 not found in new config with opid_index 1040) from {username='slave'} at 127.0.0.1:52596
I20240501 20:10:42.509481 25017 tablet_replica.cc:330] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: stopping tablet replica
I20240501 20:10:42.510928 25017 raft_consensus.cc:2229] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:42.511248 25017 raft_consensus.cc:2258] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:42.523299 25017 ts_tablet_manager.cc:1875] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:42.558009 25017 ts_tablet_manager.cc:1888] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1039
I20240501 20:10:42.558334 25017 log.cc:1196] T e4c119cb173942819ec01f7f3797c9d7 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/e4c119cb173942819ec01f7f3797c9d7
I20240501 20:10:42.559573 20164 catalog_manager.cc:4830] TS 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): tablet e4c119cb173942819ec01f7f3797c9d7 (table rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]) successfully deleted
I20240501 20:10:43.355518 21872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:43.445385 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:43.519735 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:43.545413 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:43.560660 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:43.600088 20298 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:43.642666 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:43.834188 20320 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9ccf34e292a480aa7283631edb5a0d2"
dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46632
I20240501 20:10:43.834664 20320 raft_consensus.cc:595] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Received request to transfer leadership
W20240501 20:10:43.847501 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.854676 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.860658 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.878350 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.911149 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.921751 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.930946 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.939904 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.952488 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.963783 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.978930 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:43.994273 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.011235 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.030360 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.050936 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.070268 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.092252 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.112582 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.133675 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.157454 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.181349 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
W20240501 20:10:44.207139 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
I20240501 20:10:44.229226 24477 raft_consensus.cc:979] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: : Instructing follower 133d5e0cbb234be1a5752a784d6d934a to start an election
I20240501 20:10:44.229441 24477 raft_consensus.cc:1067] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Signalling peer 133d5e0cbb234be1a5752a784d6d934a to start an election
I20240501 20:10:44.230327 21889 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9ccf34e292a480aa7283631edb5a0d2"
dest_uuid: "133d5e0cbb234be1a5752a784d6d934a"
from {username='slave'} at 127.14.106.193:37265
I20240501 20:10:44.230598 21889 raft_consensus.cc:482] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:44.230788 21889 raft_consensus.cc:3037] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:44.233605 21889 raft_consensus.cc:504] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
W20240501 20:10:44.235020 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Service unavailable: leader transfer in progress
I20240501 20:10:44.235687 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccf34e292a480aa7283631edb5a0d2" candidate_uuid: "133d5e0cbb234be1a5752a784d6d934a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1122 } } ignore_live_leader: true dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:10:44.235879 20318 raft_consensus.cc:3032] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:44.235954 20318 raft_consensus.cc:729] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Running, Role: LEADER
I20240501 20:10:44.236191 20318 consensus_queue.cc:260] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1122, Committed index: 1122, Last appended: 1.1122, Last appended by leader: 1122, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:44.236443 20318 raft_consensus.cc:3037] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:44.238143 20318 raft_consensus.cc:2454] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133d5e0cbb234be1a5752a784d6d934a in term 2.
I20240501 20:10:44.238667 21825 leader_election.cc:302] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133d5e0cbb234be1a5752a784d6d934a, 4f8d9458da0b406db99095564f9cf2ed; no voters:
I20240501 20:10:44.239233 25102 raft_consensus.cc:2781] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:44.242372 21889 leader_election.cc:288] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [CANDIDATE]: Term 2 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237), 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:10:44.242762 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccf34e292a480aa7283631edb5a0d2" candidate_uuid: "133d5e0cbb234be1a5752a784d6d934a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1122 } } ignore_live_leader: true dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:10:44.242959 20581 raft_consensus.cc:3037] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:44.243460 25102 raft_consensus.cc:686] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 2 LEADER]: Becoming Leader. State: Replica: 133d5e0cbb234be1a5752a784d6d934a, State: Running, Role: LEADER
I20240501 20:10:44.243911 25102 consensus_queue.cc:237] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1122, Committed index: 1122, Last appended: 1.1122, Last appended by leader: 1122, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:44.245725 20581 raft_consensus.cc:2454] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133d5e0cbb234be1a5752a784d6d934a in term 2.
I20240501 20:10:44.246949 20172 catalog_manager.cc:5484] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a reported cstate change: term changed from 1 to 2, leader changed from 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) to 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198). New cstate: current_term: 2 leader_uuid: "133d5e0cbb234be1a5752a784d6d934a" committed_config { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
W20240501 20:10:44.261931 20279 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42790: Illegal state: replica 4f8d9458da0b406db99095564f9cf2ed is not leader of this config: current role FOLLOWER
W20240501 20:10:44.389832 20540 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34764: Illegal state: replica 2305b26386ac446da2436463b536d296 is not leader of this config: current role FOLLOWER
I20240501 20:10:44.492460 20581 raft_consensus.cc:1261] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Refusing update from remote peer 133d5e0cbb234be1a5752a784d6d934a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1122. Preceding OpId from leader: term: 2 index: 1124. (index mismatch)
I20240501 20:10:44.492630 20320 raft_consensus.cc:1261] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Refusing update from remote peer 133d5e0cbb234be1a5752a784d6d934a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1122. Preceding OpId from leader: term: 2 index: 1124. (index mismatch)
I20240501 20:10:44.493182 25102 consensus_queue.cc:1035] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1123, Last known committed idx: 1122, Time since last communication: 0.000s
I20240501 20:10:44.493847 25102 consensus_queue.cc:1035] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1123, Last known committed idx: 1122, Time since last communication: 0.000s
I20240501 20:10:44.499871 25077 mvcc.cc:205] Tried to move back new op lower bound from 7022978025433784320 to 7022978024426622976. Current Snapshot: MvccSnapshot[applied={T|T < 7022978025433784320 or (T in {7022978025433784320})}]
I20240501 20:10:44.629976 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:44.749135 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:44.940825 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:44.979645 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:45.012233 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:45.013410 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:45.020321 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:10:45.156296 20517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:36027, user_credentials={real_user=slave}} blocked reactor thread for 71907us
I20240501 20:10:45.712409 21891 consensus_queue.cc:237] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1143, Committed index: 1143, Last appended: 2.1143, Last appended by leader: 1122, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1144 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:10:45.741045 20581 raft_consensus.cc:1261] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Refusing update from remote peer 133d5e0cbb234be1a5752a784d6d934a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1143. Preceding OpId from leader: term: 2 index: 1144. (index mismatch)
I20240501 20:10:45.744280 25131 consensus_queue.cc:1035] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1144, Last known committed idx: 1143, Time since last communication: 0.000s
I20240501 20:10:45.752581 25131 raft_consensus.cc:2932] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a [term 2 LEADER]: Committing config change with OpId 2.1144: config changed from index 855 to 1144, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 1144 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:45.758385 20172 catalog_manager.cc:5484] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a reported cstate change: config changed from index 855 to 1144, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New cstate: current_term: 2 leader_uuid: "133d5e0cbb234be1a5752a784d6d934a" committed_config { opid_index: 1144 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:45.765166 20581 raft_consensus.cc:2932] T b9ccf34e292a480aa7283631edb5a0d2 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Committing config change with OpId 2.1144: config changed from index 855 to 1144, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 1144 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } }
I20240501 20:10:45.785559 20298 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ccf34e292a480aa7283631edb5a0d2 with delete_type TABLET_DATA_TOMBSTONED (TS 4f8d9458da0b406db99095564f9cf2ed not found in new config with opid_index 1144) from {username='slave'} at 127.0.0.1:34004
I20240501 20:10:45.789754 25183 tablet_replica.cc:330] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: stopping tablet replica
I20240501 20:10:45.789991 25183 raft_consensus.cc:2229] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:45.790241 25183 raft_consensus.cc:2258] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:45.792001 25183 ts_tablet_manager.cc:1875] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:45.799993 25183 ts_tablet_manager.cc:1888] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1143
I20240501 20:10:45.800189 25183 log.cc:1196] T b9ccf34e292a480aa7283631edb5a0d2 P 4f8d9458da0b406db99095564f9cf2ed: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/b9ccf34e292a480aa7283631edb5a0d2
I20240501 20:10:45.801170 20163 catalog_manager.cc:4830] TS 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237): tablet b9ccf34e292a480aa7283631edb5a0d2 (table rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]) successfully deleted
I20240501 20:10:46.983628 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:47.020697 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:47.120731 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:47.198539 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:47.224714 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:47.255584 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:47.404549 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:48.655813 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:48.703312 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:48.705875 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:48.743961 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:48.848938 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:48.875479 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:48.921296 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:51.343231 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:51.395550 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:51.443732 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:51.456560 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:51.463559 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:51.503247 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:51.543407 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:10:52.461346 20515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:39669, user_credentials={real_user=slave}} blocked reactor thread for 56861.6us
I20240501 20:10:52.537786 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.686857 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:52.689357 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:52.775931 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:52.780735 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:52.873829 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:52.889988 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:53.655943 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:53.666276 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:53.669939 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:53.708734 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:53.720548 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:53.727205 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:53.749294 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
a92b9fd7e9164dc18d7bb7cea432762a | 127.14.106.254:36189 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.106.212:45469 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
00e1df10b53f4936b747b168af137e4b | 127.14.106.199:39669 | HEALTHY | <none> | 0 | 0
133d5e0cbb234be1a5752a784d6d934a | 127.14.106.198:43715 | HEALTHY | <none> | 1 | 0
2305b26386ac446da2436463b536d296 | 127.14.106.195:35301 | HEALTHY | <none> | 4 | 0
28b7e229ab3d4f97a09ca51ac037b008 | 127.14.106.194:39937 | HEALTHY | <none> | 5 | 0
4f8d9458da0b406db99095564f9cf2ed | 127.14.106.193:36237 | HEALTHY | <none> | 2 | 0
79e9c50adfeb44929f5d29524169c6c7 | 127.14.106.196:36027 | HEALTHY | <none> | 1 | 0
a7396c6be769436abac350f7d6e86d55 | 127.14.106.197:35605 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.14.106.193 | experimental | 127.14.106.193:36237
local_ip_for_outbound_sockets | 127.14.106.194 | experimental | 127.14.106.194:39937
local_ip_for_outbound_sockets | 127.14.106.195 | experimental | 127.14.106.195:35301
local_ip_for_outbound_sockets | 127.14.106.196 | experimental | 127.14.106.196:36027
local_ip_for_outbound_sockets | 127.14.106.197 | experimental | 127.14.106.197:35605
local_ip_for_outbound_sockets | 127.14.106.198 | experimental | 127.14.106.198:43715
local_ip_for_outbound_sockets | 127.14.106.199 | experimental | 127.14.106.199:39669
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.106.193:36237
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.106.194:39937
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.106.195:35301
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.106.196:36027
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.14.106.197:35605
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.14.106.198:43715
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.14.106.199:39669
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.106.212:45469 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:10:53.812971 14763 log_verifier.cc:126] Checking tablet 01bf239579d84c07adff90ca8f87824d
I20240501 20:10:54.284649 14763 log_verifier.cc:177] Verified matching terms for 1325 ops in tablet 01bf239579d84c07adff90ca8f87824d
I20240501 20:10:54.287264 14763 log_verifier.cc:126] Checking tablet 2e414a3678e646c6b5c6c0dad4fac30e
I20240501 20:10:54.674480 14763 log_verifier.cc:177] Verified matching terms for 1181 ops in tablet 2e414a3678e646c6b5c6c0dad4fac30e
I20240501 20:10:54.675777 14763 log_verifier.cc:126] Checking tablet 4c862c7f6ec24fe2916c97cc7c72401b
I20240501 20:10:55.093489 14763 log_verifier.cc:177] Verified matching terms for 1276 ops in tablet 4c862c7f6ec24fe2916c97cc7c72401b
I20240501 20:10:55.094805 14763 log_verifier.cc:126] Checking tablet 5a45309550b449c1bf0ba233c24140ed
I20240501 20:10:55.452673 14763 log_verifier.cc:177] Verified matching terms for 1111 ops in tablet 5a45309550b449c1bf0ba233c24140ed
I20240501 20:10:55.453819 14763 log_verifier.cc:126] Checking tablet 62a6957734c44303929840f208f5d1d3
I20240501 20:10:55.880512 14763 log_verifier.cc:177] Verified matching terms for 1309 ops in tablet 62a6957734c44303929840f208f5d1d3
I20240501 20:10:55.881958 14763 log_verifier.cc:126] Checking tablet 72363ecf8c08466b8b7f5f4a2d07e1d4
I20240501 20:10:56.277842 14763 log_verifier.cc:177] Verified matching terms for 1227 ops in tablet 72363ecf8c08466b8b7f5f4a2d07e1d4
I20240501 20:10:56.279206 14763 log_verifier.cc:126] Checking tablet 7ff0eebf61824be1aff2cf5c43f45d78
I20240501 20:10:56.717832 14763 log_verifier.cc:177] Verified matching terms for 1350 ops in tablet 7ff0eebf61824be1aff2cf5c43f45d78
I20240501 20:10:56.719419 14763 log_verifier.cc:126] Checking tablet 86d6707287cc48169ae7a548c1c01597
I20240501 20:10:57.127427 14763 log_verifier.cc:177] Verified matching terms for 1236 ops in tablet 86d6707287cc48169ae7a548c1c01597
I20240501 20:10:57.128795 14763 log_verifier.cc:126] Checking tablet afabe33672c647059a9c125c76906fac
I20240501 20:10:57.564610 14763 log_verifier.cc:177] Verified matching terms for 1301 ops in tablet afabe33672c647059a9c125c76906fac
I20240501 20:10:57.566071 14763 log_verifier.cc:126] Checking tablet b9ccf34e292a480aa7283631edb5a0d2
I20240501 20:10:57.981410 14763 log_verifier.cc:177] Verified matching terms for 1304 ops in tablet b9ccf34e292a480aa7283631edb5a0d2
I20240501 20:10:57.982877 14763 log_verifier.cc:126] Checking tablet e4c119cb173942819ec01f7f3797c9d7
I20240501 20:10:58.406158 14763 log_verifier.cc:177] Verified matching terms for 1253 ops in tablet e4c119cb173942819ec01f7f3797c9d7
I20240501 20:10:58.407488 14763 log_verifier.cc:126] Checking tablet e4dff67082a6473f994c09821a0d7a3d
I20240501 20:10:58.424885 14763 log_verifier.cc:177] Verified matching terms for 4 ops in tablet e4dff67082a6473f994c09821a0d7a3d
I20240501 20:10:58.424979 14763 log_verifier.cc:126] Checking tablet e74fb1e0e017408f946a8df8ae304f13
I20240501 20:10:58.802230 14763 log_verifier.cc:177] Verified matching terms for 1194 ops in tablet e74fb1e0e017408f946a8df8ae304f13
I20240501 20:10:58.803534 14763 log_verifier.cc:126] Checking tablet e8477d3f66b940538bb74f2cf87a3d47
I20240501 20:10:59.279753 14763 log_verifier.cc:177] Verified matching terms for 1231 ops in tablet e8477d3f66b940538bb74f2cf87a3d47
I20240501 20:10:59.281181 14763 log_verifier.cc:126] Checking tablet ee446411ba4e4da2b295080da28d40b9
I20240501 20:10:59.660530 14763 log_verifier.cc:177] Verified matching terms for 1198 ops in tablet ee446411ba4e4da2b295080da28d40b9
I20240501 20:10:59.661876 14763 log_verifier.cc:126] Checking tablet f9480642e71d46679005d4f55d2e88f8
I20240501 20:11:00.146302 14763 log_verifier.cc:177] Verified matching terms for 1407 ops in tablet f9480642e71d46679005d4f55d2e88f8
I20240501 20:11:00.156543 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.200:0
--local_ip_for_outbound_sockets=127.14.106.200
--webserver_interface=127.14.106.200
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--builtin_ntp_servers=127.14.106.212:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:11:00.537559 25522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:00.537974 25522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:00.538139 25522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:00.549360 25522 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:00.549489 25522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:00.549650 25522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.200
I20240501 20:11:00.559995 25522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.106.200
--webserver_port=0
--tserver_master_addrs=127.14.106.254:36189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.200
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:00.561391 25522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:00.563261 25522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:00.575879 25527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:00.576277 25530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:00.576689 25528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:00.577383 25522 server_base.cc:1034] running on GCE node
I20240501 20:11:00.577975 25522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:00.579085 25522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:00.580346 25522 hybrid_clock.cc:648] HybridClock initialized: now 1714594260580277 us; error 52 us; skew 500 ppm
I20240501 20:11:00.580710 25522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:00.583398 25522 webserver.cc:460] Webserver started at http://127.14.106.200:35579/ using document root <none> and password file <none>
I20240501 20:11:00.584110 25522 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:00.584208 25522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:00.584483 25522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:00.587131 25522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "cab748d6deeb4b9699877c77358fe755"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-pg4x"
I20240501 20:11:00.587862 25522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "cab748d6deeb4b9699877c77358fe755"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-pg4x"
I20240501 20:11:00.592922 25522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 20:11:00.596323 25536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:00.597630 25522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:00.597764 25522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "cab748d6deeb4b9699877c77358fe755"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-pg4x"
I20240501 20:11:00.597944 25522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:00.631793 25522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:00.632647 25522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:00.632963 25522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:00.633967 25522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:00.635433 25522 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:00.635524 25522 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:00.635607 25522 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:00.635663 25522 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:00.672004 25522 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.200:46143
I20240501 20:11:00.672080 25648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.200:46143 every 8 connection(s)
I20240501 20:11:00.673303 25522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:11:00.673727 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 25522
I20240501 20:11:00.673921 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:11:00.689159 25649 heartbeater.cc:347] Connected to a master server at 127.14.106.254:36189
I20240501 20:11:00.689497 25649 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:00.690424 25649 heartbeater.cc:511] Master 127.14.106.254:36189 requested a full tablet report, sending...
I20240501 20:11:00.692023 20175 ts_manager.cc:194] Registered new tserver with Master: cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143)
I20240501 20:11:00.693148 20175 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.200:56543
I20240501 20:11:01.192114 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:01.203084 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:01.219545 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:01.231729 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:01.253995 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:01.254407 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:01.272390 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:01.322274 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:01.472198 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:01.473526 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:01.491266 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:01.492779 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:01.494424 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:01.513859 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:01.513944 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:01.515420 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:01.635555 20319 consensus_queue.cc:237] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1227, Committed index: 1227, Last appended: 2.1227, Last appended by leader: 925, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:01.637792 21760 raft_consensus.cc:1261] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Refusing update from remote peer 4f8d9458da0b406db99095564f9cf2ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 1227. Preceding OpId from leader: term: 2 index: 1228. (index mismatch)
I20240501 20:11:01.638751 25755 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1228, Last known committed idx: 1227, Time since last communication: 0.000s
I20240501 20:11:01.641167 25349 raft_consensus.cc:2932] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 LEADER]: Committing config change with OpId 2.1228: config changed from index 945 to 1228, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New config: { opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.641649 21760 raft_consensus.cc:2932] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Committing config change with OpId 2.1228: config changed from index 945 to 1228, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New config: { opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.645812 20179 catalog_manager.cc:5484] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed reported cstate change: config changed from index 945 to 1228, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New cstate: current_term: 2 leader_uuid: "4f8d9458da0b406db99095564f9cf2ed" committed_config { opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
W20240501 20:11:01.646279 20254 consensus_peers.cc:485] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed -> Peer cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143): Couldn't send request to peer cab748d6deeb4b9699877c77358fe755. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72363ecf8c08466b8b7f5f4a2d07e1d4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:01.684136 20577 consensus_queue.cc:237] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1231, Committed index: 1231, Last appended: 1.1231, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
W20240501 20:11:01.691772 20515 consensus_peers.cc:485] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 -> Peer cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143): Couldn't send request to peer cab748d6deeb4b9699877c77358fe755. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8477d3f66b940538bb74f2cf87a3d47. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:01.694414 21890 raft_consensus.cc:1261] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Refusing update from remote peer 2305b26386ac446da2436463b536d296: Log matching property violated. Preceding OpId in replica: term: 1 index: 1231. Preceding OpId from leader: term: 1 index: 1232. (index mismatch)
I20240501 20:11:01.694922 25778 consensus_queue.cc:1035] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1232, Last known committed idx: 1231, Time since last communication: 0.000s
I20240501 20:11:01.696197 25649 heartbeater.cc:503] Master 127.14.106.254:36189 was elected leader, sending a full tablet report...
I20240501 20:11:01.697489 25440 raft_consensus.cc:2932] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Committing config change with OpId 1.1232: config changed from index 748 to 1232, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New config: { opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.699363 21890 raft_consensus.cc:2932] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Committing config change with OpId 1.1232: config changed from index 748 to 1232, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New config: { opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.704447 20174 catalog_manager.cc:5484] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 reported cstate change: config changed from index 748 to 1232, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "2305b26386ac446da2436463b536d296" committed_config { opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.736744 21760 consensus_queue.cc:237] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1111, Committed index: 1111, Last appended: 2.1111, Last appended by leader: 495, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:01.739194 20319 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Refusing update from remote peer a7396c6be769436abac350f7d6e86d55: Log matching property violated. Preceding OpId in replica: term: 2 index: 1111. Preceding OpId from leader: term: 2 index: 1112. (index mismatch)
I20240501 20:11:01.739785 25798 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1112, Last known committed idx: 1111, Time since last communication: 0.000s
I20240501 20:11:01.742581 25510 raft_consensus.cc:2932] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Committing config change with OpId 2.1112: config changed from index 510 to 1112, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New config: { opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.745378 20319 raft_consensus.cc:2932] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Committing config change with OpId 2.1112: config changed from index 510 to 1112, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New config: { opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.746896 20179 catalog_manager.cc:5484] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 reported cstate change: config changed from index 510 to 1112, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New cstate: current_term: 2 leader_uuid: "a7396c6be769436abac350f7d6e86d55" committed_config { opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
W20240501 20:11:01.754582 21693 consensus_peers.cc:485] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 -> Peer cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143): Couldn't send request to peer cab748d6deeb4b9699877c77358fe755. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a45309550b449c1bf0ba233c24140ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:01.779979 20451 consensus_queue.cc:237] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1407, Committed index: 1407, Last appended: 1.1407, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:01.782559 22021 raft_consensus.cc:1261] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Refusing update from remote peer 28b7e229ab3d4f97a09ca51ac037b008: Log matching property violated. Preceding OpId in replica: term: 1 index: 1407. Preceding OpId from leader: term: 1 index: 1408. (index mismatch)
I20240501 20:11:01.783192 25817 consensus_queue.cc:1035] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1408, Last known committed idx: 1407, Time since last communication: 0.000s
I20240501 20:11:01.786703 25401 raft_consensus.cc:2932] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Committing config change with OpId 1.1408: config changed from index 473 to 1408, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New config: { opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.787290 22021 raft_consensus.cc:2932] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Committing config change with OpId 1.1408: config changed from index 473 to 1408, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New config: { opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
W20240501 20:11:01.793118 20385 consensus_peers.cc:485] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143): Couldn't send request to peer cab748d6deeb4b9699877c77358fe755. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9480642e71d46679005d4f55d2e88f8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:01.793104 20174 catalog_manager.cc:5484] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b reported cstate change: config changed from index 473 to 1408, VOTER cab748d6deeb4b9699877c77358fe755 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "28b7e229ab3d4f97a09ca51ac037b008" committed_config { opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:01.903900 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:01.907213 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:01.911630 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:01.926913 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:01.931078 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:01.931612 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:01.945145 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:01.971026 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:02.084024 25875 ts_tablet_manager.cc:902] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: Initiating tablet copy from peer 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:11:02.086619 25875 tablet_copy_client.cc:308] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:36237
I20240501 20:11:02.094807 20340 tablet_copy_service.cc:140] P 4f8d9458da0b406db99095564f9cf2ed: Received BeginTabletCopySession request for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 from peer cab748d6deeb4b9699877c77358fe755 ({username='slave'} at 127.14.106.200:39177)
I20240501 20:11:02.095000 20340 tablet_copy_service.cc:161] P 4f8d9458da0b406db99095564f9cf2ed: Beginning new tablet copy session on tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 from peer cab748d6deeb4b9699877c77358fe755 at {username='slave'} at 127.14.106.200:39177: session id = cab748d6deeb4b9699877c77358fe755-72363ecf8c08466b8b7f5f4a2d07e1d4
I20240501 20:11:02.097080 20340 tablet_copy_source_session.cc:205] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:02.103232 25875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.116560 25875 tablet_copy_client.cc:783] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:02.116962 25875 tablet_copy_client.cc:647] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:02.122686 25875 tablet_copy_client.cc:523] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:02.127394 25875 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: Bootstrap starting.
I20240501 20:11:02.132891 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:02.150174 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:02.169224 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:02.179322 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:02.198915 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:02.205639 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:02.209843 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:02.231446 25895 ts_tablet_manager.cc:902] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: Initiating tablet copy from peer 2305b26386ac446da2436463b536d296 (127.14.106.195:35301)
I20240501 20:11:02.260764 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:02.289250 25895 tablet_copy_client.cc:308] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35301
I20240501 20:11:02.297034 20601 tablet_copy_service.cc:140] P 2305b26386ac446da2436463b536d296: Received BeginTabletCopySession request for tablet e8477d3f66b940538bb74f2cf87a3d47 from peer cab748d6deeb4b9699877c77358fe755 ({username='slave'} at 127.14.106.200:49099)
I20240501 20:11:02.297204 20601 tablet_copy_service.cc:161] P 2305b26386ac446da2436463b536d296: Beginning new tablet copy session on tablet e8477d3f66b940538bb74f2cf87a3d47 from peer cab748d6deeb4b9699877c77358fe755 at {username='slave'} at 127.14.106.200:49099: session id = cab748d6deeb4b9699877c77358fe755-e8477d3f66b940538bb74f2cf87a3d47
I20240501 20:11:02.300076 20601 tablet_copy_source_session.cc:205] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:02.301693 25895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8477d3f66b940538bb74f2cf87a3d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.309998 25895 tablet_copy_client.cc:783] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:02.310341 25895 tablet_copy_client.cc:647] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:02.315973 25895 tablet_copy_client.cc:523] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:02.320013 25895 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: Bootstrap starting.
I20240501 20:11:02.357925 25903 ts_tablet_manager.cc:902] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: Initiating tablet copy from peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
I20240501 20:11:02.361135 25902 ts_tablet_manager.cc:902] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: Initiating tablet copy from peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
I20240501 20:11:02.383136 25903 tablet_copy_client.cc:308] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.197:35605
I20240501 20:11:02.392334 21781 tablet_copy_service.cc:140] P a7396c6be769436abac350f7d6e86d55: Received BeginTabletCopySession request for tablet 5a45309550b449c1bf0ba233c24140ed from peer cab748d6deeb4b9699877c77358fe755 ({username='slave'} at 127.14.106.200:55251)
I20240501 20:11:02.392525 21781 tablet_copy_service.cc:161] P a7396c6be769436abac350f7d6e86d55: Beginning new tablet copy session on tablet 5a45309550b449c1bf0ba233c24140ed from peer cab748d6deeb4b9699877c77358fe755 at {username='slave'} at 127.14.106.200:55251: session id = cab748d6deeb4b9699877c77358fe755-5a45309550b449c1bf0ba233c24140ed
I20240501 20:11:02.395507 21781 tablet_copy_source_session.cc:205] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:02.395510 25902 tablet_copy_client.cc:308] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:39937
I20240501 20:11:02.397205 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a45309550b449c1bf0ba233c24140ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.412362 20471 tablet_copy_service.cc:140] P 28b7e229ab3d4f97a09ca51ac037b008: Received BeginTabletCopySession request for tablet f9480642e71d46679005d4f55d2e88f8 from peer cab748d6deeb4b9699877c77358fe755 ({username='slave'} at 127.14.106.200:40277)
I20240501 20:11:02.412521 20471 tablet_copy_service.cc:161] P 28b7e229ab3d4f97a09ca51ac037b008: Beginning new tablet copy session on tablet f9480642e71d46679005d4f55d2e88f8 from peer cab748d6deeb4b9699877c77358fe755 at {username='slave'} at 127.14.106.200:40277: session id = cab748d6deeb4b9699877c77358fe755-f9480642e71d46679005d4f55d2e88f8
I20240501 20:11:02.415470 20471 tablet_copy_source_session.cc:205] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:02.416913 25903 tablet_copy_client.cc:783] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:02.417037 25902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9480642e71d46679005d4f55d2e88f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.417282 25903 tablet_copy_client.cc:647] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:02.422669 25903 tablet_copy_client.cc:523] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:02.424381 25902 tablet_copy_client.cc:783] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:02.424692 25902 tablet_copy_client.cc:647] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:02.431823 25902 tablet_copy_client.cc:523] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:02.432852 25903 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: Bootstrap starting.
I20240501 20:11:02.436607 25902 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: Bootstrap starting.
I20240501 20:11:02.519783 25875 log.cc:824] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:02.860019 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:02.877061 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:02.878885 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:02.894737 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:02.961956 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:02.976967 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:03.022346 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:03.038069 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:03.395023 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:03.399650 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:03.455300 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:03.457973 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:03.460330 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:03.474792 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:03.528396 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:03.646998 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:04.243824 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:04.246682 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:04.258571 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:04.265195 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:04.275128 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:04.362162 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:11:04.415184 20254 consensus_peers.cc:485] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed -> Peer cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143): Couldn't send request to peer cab748d6deeb4b9699877c77358fe755. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:04.432063 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:04.506016 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:11:04.615339 20515 consensus_peers.cc:485] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 -> Peer cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143): Couldn't send request to peer cab748d6deeb4b9699877c77358fe755. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:04.935220 20385 consensus_peers.cc:485] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 -> Peer cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143): Couldn't send request to peer cab748d6deeb4b9699877c77358fe755. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:04.966591 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:04.973701 25903 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: Bootstrap replayed 1/1 log segments. Stats: ops{read=1112 overwritten=0 applied=1112 ignored=0} inserts{seen=1740 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:04.974571 25903 tablet_bootstrap.cc:492] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: Bootstrap complete.
I20240501 20:11:04.975356 25903 ts_tablet_manager.cc:1367] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: Time spent bootstrapping tablet: real 2.543s user 1.301s sys 0.104s
I20240501 20:11:04.982452 25903 raft_consensus.cc:348] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:04.983166 25903 raft_consensus.cc:729] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab748d6deeb4b9699877c77358fe755, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.983989 25903 consensus_queue.cc:260] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1112, Last appended: 2.1112, Last appended by leader: 1112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
W20240501 20:11:05.024955 21693 consensus_peers.cc:485] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 -> Peer cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143): Couldn't send request to peer cab748d6deeb4b9699877c77358fe755. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:05.031005 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:05.037803 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:05.043227 25903 ts_tablet_manager.cc:1398] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755: Time spent starting tablet: real 0.068s user 0.020s sys 0.041s
I20240501 20:11:05.048041 21781 tablet_copy_service.cc:342] P a7396c6be769436abac350f7d6e86d55: Request end of tablet copy session cab748d6deeb4b9699877c77358fe755-5a45309550b449c1bf0ba233c24140ed received from {username='slave'} at 127.14.106.200:55251
I20240501 20:11:05.048166 21781 tablet_copy_service.cc:433] P a7396c6be769436abac350f7d6e86d55: ending tablet copy session cab748d6deeb4b9699877c77358fe755-5a45309550b449c1bf0ba233c24140ed on tablet 5a45309550b449c1bf0ba233c24140ed with peer cab748d6deeb4b9699877c77358fe755
I20240501 20:11:05.104323 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:05.112521 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:05.152032 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:05.157649 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:05.169488 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:05.230420 25875 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: Bootstrap replayed 1/1 log segments. Stats: ops{read=1228 overwritten=0 applied=1228 ignored=0} inserts{seen=1897 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:05.231051 25875 tablet_bootstrap.cc:492] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: Bootstrap complete.
I20240501 20:11:05.231552 25875 ts_tablet_manager.cc:1367] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: Time spent bootstrapping tablet: real 3.105s user 1.424s sys 0.127s
I20240501 20:11:05.232704 25875 raft_consensus.cc:348] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:05.233034 25875 raft_consensus.cc:729] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab748d6deeb4b9699877c77358fe755, State: Initialized, Role: FOLLOWER
I20240501 20:11:05.233497 25875 consensus_queue.cc:260] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1228, Last appended: 2.1228, Last appended by leader: 1228, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:05.241950 25875 ts_tablet_manager.cc:1398] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:11:05.242785 20340 tablet_copy_service.cc:342] P 4f8d9458da0b406db99095564f9cf2ed: Request end of tablet copy session cab748d6deeb4b9699877c77358fe755-72363ecf8c08466b8b7f5f4a2d07e1d4 received from {username='slave'} at 127.14.106.200:39177
I20240501 20:11:05.242893 20340 tablet_copy_service.cc:433] P 4f8d9458da0b406db99095564f9cf2ed: ending tablet copy session cab748d6deeb4b9699877c77358fe755-72363ecf8c08466b8b7f5f4a2d07e1d4 on tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 with peer cab748d6deeb4b9699877c77358fe755
I20240501 20:11:05.249199 25895 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: Bootstrap replayed 1/1 log segments. Stats: ops{read=1232 overwritten=0 applied=1232 ignored=0} inserts{seen=1915 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:05.250355 25895 tablet_bootstrap.cc:492] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: Bootstrap complete.
I20240501 20:11:05.251124 25895 ts_tablet_manager.cc:1367] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: Time spent bootstrapping tablet: real 2.931s user 1.456s sys 0.122s
I20240501 20:11:05.259951 25895 raft_consensus.cc:348] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:05.260320 25895 raft_consensus.cc:729] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab748d6deeb4b9699877c77358fe755, State: Initialized, Role: FOLLOWER
I20240501 20:11:05.260624 25895 consensus_queue.cc:260] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1232, Last appended: 1.1232, Last appended by leader: 1232, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:05.261273 25895 ts_tablet_manager.cc:1398] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 20:11:05.262475 20601 tablet_copy_service.cc:342] P 2305b26386ac446da2436463b536d296: Request end of tablet copy session cab748d6deeb4b9699877c77358fe755-e8477d3f66b940538bb74f2cf87a3d47 received from {username='slave'} at 127.14.106.200:49099
I20240501 20:11:05.262614 20601 tablet_copy_service.cc:433] P 2305b26386ac446da2436463b536d296: ending tablet copy session cab748d6deeb4b9699877c77358fe755-e8477d3f66b940538bb74f2cf87a3d47 on tablet e8477d3f66b940538bb74f2cf87a3d47 with peer cab748d6deeb4b9699877c77358fe755
I20240501 20:11:05.367715 25603 raft_consensus.cc:1203] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1227->[2.1228-2.1228] Dedup: 2.1228->[]
I20240501 20:11:05.420837 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:05.421713 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:05.435781 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:05.439258 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:05.447743 25603 raft_consensus.cc:1203] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1111->[2.1112-2.1112] Dedup: 2.1112->[]
I20240501 20:11:05.450812 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:05.531366 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:05.532881 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:05.544648 25902 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: Bootstrap replayed 1/1 log segments. Stats: ops{read=1408 overwritten=0 applied=1408 ignored=0} inserts{seen=2078 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:05.545164 25902 tablet_bootstrap.cc:492] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: Bootstrap complete.
I20240501 20:11:05.545629 25902 ts_tablet_manager.cc:1367] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: Time spent bootstrapping tablet: real 3.109s user 1.594s sys 0.133s
I20240501 20:11:05.546427 25902 raft_consensus.cc:348] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:05.546717 25902 raft_consensus.cc:729] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab748d6deeb4b9699877c77358fe755, State: Initialized, Role: FOLLOWER
I20240501 20:11:05.546996 25902 consensus_queue.cc:260] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1408, Last appended: 1.1408, Last appended by leader: 1408, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:05.547577 25902 ts_tablet_manager.cc:1398] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:05.549263 20471 tablet_copy_service.cc:342] P 28b7e229ab3d4f97a09ca51ac037b008: Request end of tablet copy session cab748d6deeb4b9699877c77358fe755-f9480642e71d46679005d4f55d2e88f8 received from {username='slave'} at 127.14.106.200:40277
I20240501 20:11:05.549387 20471 tablet_copy_service.cc:433] P 28b7e229ab3d4f97a09ca51ac037b008: ending tablet copy session cab748d6deeb4b9699877c77358fe755-f9480642e71d46679005d4f55d2e88f8 on tablet f9480642e71d46679005d4f55d2e88f8 with peer cab748d6deeb4b9699877c77358fe755
I20240501 20:11:05.555785 25603 raft_consensus.cc:1203] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1407->[1.1408-1.1408] Dedup: 1.1408->[]
I20240501 20:11:05.577724 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:05.627553 25603 raft_consensus.cc:1203] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1231->[1.1232-1.1232] Dedup: 1.1232->[]
I20240501 20:11:05.747503 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:05.751672 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:05.751775 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:05.754971 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:05.758843 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:05.771554 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:05.772255 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:05.801465 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:06.179185 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.181480 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:06.201192 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:06.217621 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:06.243342 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:06.290838 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.310978 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:06.311731 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:06.375946 20451 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9480642e71d46679005d4f55d2e88f8"
dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34830
I20240501 20:11:06.376124 20451 raft_consensus.cc:595] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:06.479279 22002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:06.479686 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.481979 20431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:06.500324 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:06.500324 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.510982 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:06.515336 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:06.515643 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:06.615075 26251 raft_consensus.cc:979] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: : Instructing follower 00e1df10b53f4936b747b168af137e4b to start an election
I20240501 20:11:06.615242 26251 raft_consensus.cc:1067] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Signalling peer 00e1df10b53f4936b747b168af137e4b to start an election
I20240501 20:11:06.615922 22021 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9480642e71d46679005d4f55d2e88f8"
dest_uuid: "00e1df10b53f4936b747b168af137e4b"
from {username='slave'} at 127.14.106.194:54853
I20240501 20:11:06.616216 22021 raft_consensus.cc:482] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.616346 22021 raft_consensus.cc:3037] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.619159 22021 raft_consensus.cc:504] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:06.620702 22021 leader_election.cc:288] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [CANDIDATE]: Term 2 election: Requested vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937), cab748d6deeb4b9699877c77358fe755 (127.14.106.200:46143)
I20240501 20:11:06.628739 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9480642e71d46679005d4f55d2e88f8" candidate_uuid: "00e1df10b53f4936b747b168af137e4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1408 } } ignore_live_leader: true dest_uuid: "28b7e229ab3d4f97a09ca51ac037b008"
I20240501 20:11:06.628945 20451 raft_consensus.cc:3032] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.629019 20451 raft_consensus.cc:729] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28b7e229ab3d4f97a09ca51ac037b008, State: Running, Role: LEADER
I20240501 20:11:06.629256 20451 consensus_queue.cc:260] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1408, Committed index: 1408, Last appended: 1.1408, Last appended by leader: 1408, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:06.629668 20451 raft_consensus.cc:3037] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.630553 25603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9480642e71d46679005d4f55d2e88f8" candidate_uuid: "00e1df10b53f4936b747b168af137e4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1408 } } ignore_live_leader: true dest_uuid: "cab748d6deeb4b9699877c77358fe755"
I20240501 20:11:06.630895 25603 raft_consensus.cc:3037] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.632700 20451 raft_consensus.cc:2454] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e1df10b53f4936b747b168af137e4b in term 2.
I20240501 20:11:06.633562 25603 raft_consensus.cc:2454] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e1df10b53f4936b747b168af137e4b in term 2.
I20240501 20:11:06.634058 21956 leader_election.cc:302] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e1df10b53f4936b747b168af137e4b, cab748d6deeb4b9699877c77358fe755; no voters:
I20240501 20:11:06.634771 26260 raft_consensus.cc:2781] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.635313 26260 raft_consensus.cc:686] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 2 LEADER]: Becoming Leader. State: Replica: 00e1df10b53f4936b747b168af137e4b, State: Running, Role: LEADER
I20240501 20:11:06.635789 26260 consensus_queue.cc:237] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1408, Committed index: 1408, Last appended: 1.1408, Last appended by leader: 1408, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:06.638736 20171 catalog_manager.cc:5484] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b reported cstate change: term changed from 1 to 2, leader changed from 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) to 00e1df10b53f4936b747b168af137e4b (127.14.106.199). New cstate: current_term: 2 leader_uuid: "00e1df10b53f4936b747b168af137e4b" committed_config { opid_index: 1408 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:06.727562 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.728519 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.740872 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:06.747822 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:06.752394 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:06.785914 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.791028 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:06.824548 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:06.869406 21760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed"
dest_uuid: "a7396c6be769436abac350f7d6e86d55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34292
I20240501 20:11:06.869616 21760 raft_consensus.cc:595] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Received request to transfer leadership
I20240501 20:11:06.976200 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.979740 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.985664 21740 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:06.985836 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:06.987691 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:06.989780 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:06.995260 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:07.035491 25584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.102212 26333 raft_consensus.cc:979] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: : Instructing follower cab748d6deeb4b9699877c77358fe755 to start an election
I20240501 20:11:07.102416 26333 raft_consensus.cc:1067] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Signalling peer cab748d6deeb4b9699877c77358fe755 to start an election
I20240501 20:11:07.103500 25603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed"
dest_uuid: "cab748d6deeb4b9699877c77358fe755"
from {username='slave'} at 127.14.106.197:53319
I20240501 20:11:07.103921 25603 raft_consensus.cc:482] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.104084 25603 raft_consensus.cc:3037] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.106256 25603 raft_consensus.cc:504] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:07.107825 25603 leader_election.cc:288] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [CANDIDATE]: Term 3 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237), a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
I20240501 20:11:07.108283 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed" candidate_uuid: "cab748d6deeb4b9699877c77358fe755" candidate_term: 3 candidate_status { last_received { term: 2 index: 1112 } } ignore_live_leader: true dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:11:07.108510 20319 raft_consensus.cc:3037] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.108592 21760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a45309550b449c1bf0ba233c24140ed" candidate_uuid: "cab748d6deeb4b9699877c77358fe755" candidate_term: 3 candidate_status { last_received { term: 2 index: 1112 } } ignore_live_leader: true dest_uuid: "a7396c6be769436abac350f7d6e86d55"
I20240501 20:11:07.108814 21760 raft_consensus.cc:3032] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.108942 21760 raft_consensus.cc:729] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a7396c6be769436abac350f7d6e86d55, State: Running, Role: LEADER
I20240501 20:11:07.109261 21760 consensus_queue.cc:260] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1112, Committed index: 1112, Last appended: 2.1112, Last appended by leader: 1112, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:07.109683 21760 raft_consensus.cc:3037] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.111169 20319 raft_consensus.cc:2454] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab748d6deeb4b9699877c77358fe755 in term 3.
I20240501 20:11:07.111646 21760 raft_consensus.cc:2454] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab748d6deeb4b9699877c77358fe755 in term 3.
I20240501 20:11:07.111773 25537 leader_election.cc:302] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9458da0b406db99095564f9cf2ed, cab748d6deeb4b9699877c77358fe755; no voters:
I20240501 20:11:07.112643 26334 raft_consensus.cc:2781] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.113292 26334 raft_consensus.cc:686] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 3 LEADER]: Becoming Leader. State: Replica: cab748d6deeb4b9699877c77358fe755, State: Running, Role: LEADER
I20240501 20:11:07.113903 26334 consensus_queue.cc:237] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1112, Committed index: 1112, Last appended: 2.1112, Last appended by leader: 1112, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:07.118115 20174 catalog_manager.cc:5484] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 reported cstate change: term changed from 2 to 3, leader changed from a7396c6be769436abac350f7d6e86d55 (127.14.106.197) to cab748d6deeb4b9699877c77358fe755 (127.14.106.200). New cstate: current_term: 3 leader_uuid: "cab748d6deeb4b9699877c77358fe755" committed_config { opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:07.147120 20451 raft_consensus.cc:1261] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Refusing update from remote peer 00e1df10b53f4936b747b168af137e4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1408. Preceding OpId from leader: term: 2 index: 1409. (index mismatch)
I20240501 20:11:07.147814 26347 consensus_queue.cc:1035] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1409, Last known committed idx: 1408, Time since last communication: 0.000s
I20240501 20:11:07.167882 25603 raft_consensus.cc:1261] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Refusing update from remote peer 00e1df10b53f4936b747b168af137e4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1408. Preceding OpId from leader: term: 2 index: 1409. (index mismatch)
I20240501 20:11:07.168473 26360 consensus_queue.cc:1035] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1409, Last known committed idx: 1408, Time since last communication: 0.000s
I20240501 20:11:07.256978 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.257247 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:07.278580 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.289386 20298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:07.308408 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:07.339789 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:07.341054 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.373586 25584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.427531 20319 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4"
dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54986
I20240501 20:11:07.427711 20319 raft_consensus.cc:595] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 LEADER]: Received request to transfer leadership
I20240501 20:11:07.519882 26417 raft_consensus.cc:979] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: : Instructing follower cab748d6deeb4b9699877c77358fe755 to start an election
I20240501 20:11:07.520198 26118 raft_consensus.cc:1067] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 LEADER]: Signalling peer cab748d6deeb4b9699877c77358fe755 to start an election
I20240501 20:11:07.520949 25603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4"
dest_uuid: "cab748d6deeb4b9699877c77358fe755"
from {username='slave'} at 127.14.106.193:56531
I20240501 20:11:07.521210 25603 raft_consensus.cc:482] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.521327 25603 raft_consensus.cc:3037] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.523666 25603 raft_consensus.cc:504] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:07.525547 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4" candidate_uuid: "cab748d6deeb4b9699877c77358fe755" candidate_term: 3 candidate_status { last_received { term: 2 index: 1228 } } ignore_live_leader: true dest_uuid: "4f8d9458da0b406db99095564f9cf2ed"
I20240501 20:11:07.525956 25603 leader_election.cc:288] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [CANDIDATE]: Term 3 election: Requested vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237), a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605)
I20240501 20:11:07.526038 21760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72363ecf8c08466b8b7f5f4a2d07e1d4" candidate_uuid: "cab748d6deeb4b9699877c77358fe755" candidate_term: 3 candidate_status { last_received { term: 2 index: 1228 } } ignore_live_leader: true dest_uuid: "a7396c6be769436abac350f7d6e86d55"
I20240501 20:11:07.526232 21760 raft_consensus.cc:3037] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.526522 20319 raft_consensus.cc:3032] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.526605 20319 raft_consensus.cc:729] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4f8d9458da0b406db99095564f9cf2ed, State: Running, Role: LEADER
I20240501 20:11:07.526856 20319 consensus_queue.cc:260] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1228, Committed index: 1228, Last appended: 2.1228, Last appended by leader: 1228, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:07.527232 20319 raft_consensus.cc:3037] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.528556 21760 raft_consensus.cc:2454] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab748d6deeb4b9699877c77358fe755 in term 3.
I20240501 20:11:07.529021 25540 leader_election.cc:302] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7396c6be769436abac350f7d6e86d55, cab748d6deeb4b9699877c77358fe755; no voters:
I20240501 20:11:07.529340 26334 raft_consensus.cc:2781] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.529551 26334 raft_consensus.cc:686] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 3 LEADER]: Becoming Leader. State: Replica: cab748d6deeb4b9699877c77358fe755, State: Running, Role: LEADER
I20240501 20:11:07.529796 26334 consensus_queue.cc:237] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1228, Committed index: 1228, Last appended: 2.1228, Last appended by leader: 1228, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:07.530992 20319 raft_consensus.cc:2454] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab748d6deeb4b9699877c77358fe755 in term 3.
I20240501 20:11:07.535657 21760 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 3 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 2 index: 1112. Preceding OpId from leader: term: 3 index: 1113. (index mismatch)
I20240501 20:11:07.536455 26335 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1113, Last known committed idx: 1112, Time since last communication: 0.000s
I20240501 20:11:07.543738 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.544437 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.547493 20319 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 3 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 2 index: 1112. Preceding OpId from leader: term: 3 index: 1113. (index mismatch)
I20240501 20:11:07.547909 20175 catalog_manager.cc:5484] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 reported cstate change: term changed from 2 to 3, leader changed from 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) to cab748d6deeb4b9699877c77358fe755 (127.14.106.200). New cstate: current_term: 3 leader_uuid: "cab748d6deeb4b9699877c77358fe755" committed_config { opid_index: 1228 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:07.549293 26423 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1113, Last known committed idx: 1112, Time since last communication: 0.000s
I20240501 20:11:07.575417 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:07.597504 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:07.625411 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.642678 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:07.644054 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.654371 25584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:07.829536 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.834371 20560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:07.843819 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.844761 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.845635 25584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:07.855090 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:07.861208 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.880560 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:07.948112 20319 raft_consensus.cc:1261] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 3 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 2 index: 1228. Preceding OpId from leader: term: 3 index: 1229. (index mismatch)
I20240501 20:11:07.948837 26423 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1229, Last known committed idx: 1228, Time since last communication: 0.000s
I20240501 20:11:07.952719 21760 raft_consensus.cc:1261] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 3 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 2 index: 1228. Preceding OpId from leader: term: 3 index: 1229. (index mismatch)
I20240501 20:11:07.954782 26423 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1229, Last known committed idx: 1228, Time since last communication: 0.000s
I20240501 20:11:07.972939 20577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8477d3f66b940538bb74f2cf87a3d47"
dest_uuid: "2305b26386ac446da2436463b536d296"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36600
I20240501 20:11:07.973143 20577 raft_consensus.cc:595] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:08.015839 26486 raft_consensus.cc:979] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: : Instructing follower cab748d6deeb4b9699877c77358fe755 to start an election
I20240501 20:11:08.015988 26486 raft_consensus.cc:1067] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Signalling peer cab748d6deeb4b9699877c77358fe755 to start an election
I20240501 20:11:08.016645 25603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8477d3f66b940538bb74f2cf87a3d47"
dest_uuid: "cab748d6deeb4b9699877c77358fe755"
from {username='slave'} at 127.14.106.195:42331
I20240501 20:11:08.016853 25603 raft_consensus.cc:482] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.016953 25603 raft_consensus.cc:3037] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:08.019104 25603 raft_consensus.cc:504] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:08.019896 25603 leader_election.cc:288] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [CANDIDATE]: Term 2 election: Requested vote from peers 2305b26386ac446da2436463b536d296 (127.14.106.195:35301), 133d5e0cbb234be1a5752a784d6d934a (127.14.106.198:43715)
I20240501 20:11:08.020999 20577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8477d3f66b940538bb74f2cf87a3d47" candidate_uuid: "cab748d6deeb4b9699877c77358fe755" candidate_term: 2 candidate_status { last_received { term: 1 index: 1232 } } ignore_live_leader: true dest_uuid: "2305b26386ac446da2436463b536d296"
I20240501 20:11:08.021240 20577 raft_consensus.cc:3032] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:08.021319 20577 raft_consensus.cc:729] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2305b26386ac446da2436463b536d296, State: Running, Role: LEADER
I20240501 20:11:08.021574 20577 consensus_queue.cc:260] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1232, Committed index: 1232, Last appended: 1.1232, Last appended by leader: 1232, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:08.021986 20577 raft_consensus.cc:3037] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:08.024821 20577 raft_consensus.cc:2454] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab748d6deeb4b9699877c77358fe755 in term 2.
I20240501 20:11:08.025501 25538 leader_election.cc:302] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2305b26386ac446da2436463b536d296, cab748d6deeb4b9699877c77358fe755; no voters:
I20240501 20:11:08.025848 26423 raft_consensus.cc:2781] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:08.026000 26423 raft_consensus.cc:686] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 2 LEADER]: Becoming Leader. State: Replica: cab748d6deeb4b9699877c77358fe755, State: Running, Role: LEADER
I20240501 20:11:08.026348 26423 consensus_queue.cc:237] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1232, Committed index: 1232, Last appended: 1.1232, Last appended by leader: 1232, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:08.028800 21890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8477d3f66b940538bb74f2cf87a3d47" candidate_uuid: "cab748d6deeb4b9699877c77358fe755" candidate_term: 2 candidate_status { last_received { term: 1 index: 1232 } } ignore_live_leader: true dest_uuid: "133d5e0cbb234be1a5752a784d6d934a"
I20240501 20:11:08.029173 21890 raft_consensus.cc:3037] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:08.032832 21890 raft_consensus.cc:2454] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab748d6deeb4b9699877c77358fe755 in term 2.
I20240501 20:11:08.036375 20175 catalog_manager.cc:5484] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 reported cstate change: term changed from 1 to 2, leader changed from 2305b26386ac446da2436463b536d296 (127.14.106.195) to cab748d6deeb4b9699877c77358fe755 (127.14.106.200). New cstate: current_term: 2 leader_uuid: "cab748d6deeb4b9699877c77358fe755" committed_config { opid_index: 1232 OBSOLETE_local: false peers { permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:08.104315 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.117911 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:08.126747 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.129256 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.130125 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.136320 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:08.139282 25584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:08.161263 20560 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:08.502204 21890 raft_consensus.cc:1261] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 2 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 1 index: 1232. Preceding OpId from leader: term: 2 index: 1233. (index mismatch)
I20240501 20:11:08.502846 26423 consensus_queue.cc:1035] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1233, Last known committed idx: 1232, Time since last communication: 0.000s
I20240501 20:11:08.523450 20577 raft_consensus.cc:1261] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 1 index: 1232. Preceding OpId from leader: term: 2 index: 1233. (index mismatch)
I20240501 20:11:08.524401 26423 consensus_queue.cc:1035] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2305b26386ac446da2436463b536d296" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1233, Last known committed idx: 1232, Time since last communication: 0.000s
I20240501 20:11:08.595713 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.596863 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.600405 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.607854 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:08.616528 25584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:08.635380 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.671289 20560 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:08.683801 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:08.760794 22021 consensus_queue.cc:237] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1409, Committed index: 1409, Last appended: 2.1409, Last appended by leader: 1408, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1410 OBSOLETE_local: false peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:08.762758 25603 raft_consensus.cc:1261] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Refusing update from remote peer 00e1df10b53f4936b747b168af137e4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1409. Preceding OpId from leader: term: 2 index: 1410. (index mismatch)
I20240501 20:11:08.763319 26360 consensus_queue.cc:1035] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1410, Last known committed idx: 1409, Time since last communication: 0.000s
I20240501 20:11:08.766222 26578 raft_consensus.cc:2932] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b [term 2 LEADER]: Committing config change with OpId 2.1410: config changed from index 1408 to 1410, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 1410 OBSOLETE_local: false peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:08.766995 25603 raft_consensus.cc:2932] T f9480642e71d46679005d4f55d2e88f8 P cab748d6deeb4b9699877c77358fe755 [term 2 FOLLOWER]: Committing config change with OpId 2.1410: config changed from index 1408 to 1410, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New config: { opid_index: 1410 OBSOLETE_local: false peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:08.772389 20175 catalog_manager.cc:5484] T f9480642e71d46679005d4f55d2e88f8 P 00e1df10b53f4936b747b168af137e4b reported cstate change: config changed from index 1408 to 1410, VOTER 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "00e1df10b53f4936b747b168af137e4b" committed_config { opid_index: 1410 OBSOLETE_local: false peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:08.776824 20431 tablet_service.cc:1514] Processing DeleteTablet for tablet f9480642e71d46679005d4f55d2e88f8 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7e229ab3d4f97a09ca51ac037b008 not found in new config with opid_index 1410) from {username='slave'} at 127.0.0.1:52596
I20240501 20:11:08.777426 26584 tablet_replica.cc:330] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: stopping tablet replica
I20240501 20:11:08.777659 26584 raft_consensus.cc:2229] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:08.777814 26584 raft_consensus.cc:2258] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:08.779570 26584 ts_tablet_manager.cc:1875] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:08.785044 26584 ts_tablet_manager.cc:1888] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1409
I20240501 20:11:08.785143 26584 log.cc:1196] T f9480642e71d46679005d4f55d2e88f8 P 28b7e229ab3d4f97a09ca51ac037b008: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9480642e71d46679005d4f55d2e88f8
I20240501 20:11:08.785910 20164 catalog_manager.cc:4830] TS 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): tablet f9480642e71d46679005d4f55d2e88f8 (table rebalance_test_table_1 [id=da3676c5753a4bcc8f3d75f8dec89f70]) successfully deleted
I20240501 20:11:08.874193 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.881336 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.881615 20560 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:08.899439 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.902551 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:08.946601 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:08.961649 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:08.986882 25584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:09.080298 25603 consensus_queue.cc:237] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1113, Committed index: 1113, Last appended: 3.1113, Last appended by leader: 1112, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1114 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:09.082370 20319 raft_consensus.cc:1261] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 3 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 3 index: 1113. Preceding OpId from leader: term: 3 index: 1114. (index mismatch)
I20240501 20:11:09.082918 26424 consensus_queue.cc:1035] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1114, Last known committed idx: 1113, Time since last communication: 0.000s
I20240501 20:11:09.085409 26640 raft_consensus.cc:2932] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 [term 3 LEADER]: Committing config change with OpId 3.1114: config changed from index 1112 to 1114, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) evicted. New config: { opid_index: 1114 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.085880 20319 raft_consensus.cc:2932] T 5a45309550b449c1bf0ba233c24140ed P 4f8d9458da0b406db99095564f9cf2ed [term 3 FOLLOWER]: Committing config change with OpId 3.1114: config changed from index 1112 to 1114, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) evicted. New config: { opid_index: 1114 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.090181 20171 catalog_manager.cc:5484] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 reported cstate change: config changed from index 1112 to 1114, VOTER a7396c6be769436abac350f7d6e86d55 (127.14.106.197) evicted. New cstate: current_term: 3 leader_uuid: "cab748d6deeb4b9699877c77358fe755" committed_config { opid_index: 1114 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.099020 21740 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a45309550b449c1bf0ba233c24140ed with delete_type TABLET_DATA_TOMBSTONED (TS a7396c6be769436abac350f7d6e86d55 not found in new config with opid_index 1114) from {username='slave'} at 127.0.0.1:34388
I20240501 20:11:09.099849 26647 tablet_replica.cc:330] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: stopping tablet replica
I20240501 20:11:09.100320 26647 raft_consensus.cc:2229] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:09.100569 26647 raft_consensus.cc:2258] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:09.102557 26647 ts_tablet_manager.cc:1875] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:09.111737 26647 ts_tablet_manager.cc:1888] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1113
I20240501 20:11:09.111929 26647 log.cc:1196] T 5a45309550b449c1bf0ba233c24140ed P a7396c6be769436abac350f7d6e86d55: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/wals/5a45309550b449c1bf0ba233c24140ed
I20240501 20:11:09.112737 20166 catalog_manager.cc:4830] TS a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): tablet 5a45309550b449c1bf0ba233c24140ed (table rebalance_test_table_3 [id=9e0813398d56456bb972f8bd0cebde2c]) successfully deleted
I20240501 20:11:09.224414 20560 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:09.225168 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:09.225234 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:09.227763 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:09.244794 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:09.262687 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:09.268920 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:09.292943 25584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:09.387256 25603 consensus_queue.cc:237] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1229, Committed index: 1229, Last appended: 3.1229, Last appended by leader: 1228, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1230 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:09.389016 21760 raft_consensus.cc:1261] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 3 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 3 index: 1229. Preceding OpId from leader: term: 3 index: 1230. (index mismatch)
I20240501 20:11:09.389550 26640 consensus_queue.cc:1035] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1230, Last known committed idx: 1229, Time since last communication: 0.000s
I20240501 20:11:09.391813 26424 raft_consensus.cc:2932] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 [term 3 LEADER]: Committing config change with OpId 3.1230: config changed from index 1228 to 1230, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 1230 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.392305 21760 raft_consensus.cc:2932] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P a7396c6be769436abac350f7d6e86d55 [term 3 FOLLOWER]: Committing config change with OpId 3.1230: config changed from index 1228 to 1230, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New config: { opid_index: 1230 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.396064 20174 catalog_manager.cc:5484] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 reported cstate change: config changed from index 1228 to 1230, VOTER 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193) evicted. New cstate: current_term: 3 leader_uuid: "cab748d6deeb4b9699877c77358fe755" committed_config { opid_index: 1230 OBSOLETE_local: false peers { permanent_uuid: "a7396c6be769436abac350f7d6e86d55" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 35605 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.399562 20298 tablet_service.cc:1514] Processing DeleteTablet for tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 with delete_type TABLET_DATA_TOMBSTONED (TS 4f8d9458da0b406db99095564f9cf2ed not found in new config with opid_index 1230) from {username='slave'} at 127.0.0.1:34004
I20240501 20:11:09.400300 26699 tablet_replica.cc:330] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: stopping tablet replica
I20240501 20:11:09.400508 26699 raft_consensus.cc:2229] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:09.400815 26699 raft_consensus.cc:2258] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:09.402545 26699 ts_tablet_manager.cc:1875] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:09.408540 26699 ts_tablet_manager.cc:1888] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1229
I20240501 20:11:09.408672 26699 log.cc:1196] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P 4f8d9458da0b406db99095564f9cf2ed: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/72363ecf8c08466b8b7f5f4a2d07e1d4
I20240501 20:11:09.409509 20163 catalog_manager.cc:4830] TS 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237): tablet 72363ecf8c08466b8b7f5f4a2d07e1d4 (table rebalance_test_table_4 [id=e01c0d42a3d0401d8e5c256e5a7a37f4]) successfully deleted
I20240501 20:11:09.486588 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:09.509006 20560 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:09.514741 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:09.518257 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:09.523634 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:09.539739 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:09.543560 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:09.553364 25584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:09.662858 25603 consensus_queue.cc:237] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1233, Committed index: 1233, Last appended: 2.1233, Last appended by leader: 1232, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1234 OBSOLETE_local: false peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } }
I20240501 20:11:09.664661 21890 raft_consensus.cc:1261] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 2 FOLLOWER]: Refusing update from remote peer cab748d6deeb4b9699877c77358fe755: Log matching property violated. Preceding OpId in replica: term: 2 index: 1233. Preceding OpId from leader: term: 2 index: 1234. (index mismatch)
I20240501 20:11:09.665179 26424 consensus_queue.cc:1035] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1234, Last known committed idx: 1233, Time since last communication: 0.000s
I20240501 20:11:09.667587 26640 raft_consensus.cc:2932] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 [term 2 LEADER]: Committing config change with OpId 2.1234: config changed from index 1232 to 1234, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 1234 OBSOLETE_local: false peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.668042 21890 raft_consensus.cc:2932] T e8477d3f66b940538bb74f2cf87a3d47 P 133d5e0cbb234be1a5752a784d6d934a [term 2 FOLLOWER]: Committing config change with OpId 2.1234: config changed from index 1232 to 1234, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New config: { opid_index: 1234 OBSOLETE_local: false peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.672142 20172 catalog_manager.cc:5484] T e8477d3f66b940538bb74f2cf87a3d47 P cab748d6deeb4b9699877c77358fe755 reported cstate change: config changed from index 1232 to 1234, VOTER 2305b26386ac446da2436463b536d296 (127.14.106.195) evicted. New cstate: current_term: 2 leader_uuid: "cab748d6deeb4b9699877c77358fe755" committed_config { opid_index: 1234 OBSOLETE_local: false peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } } peers { permanent_uuid: "cab748d6deeb4b9699877c77358fe755" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 46143 } } }
I20240501 20:11:09.679066 20560 tablet_service.cc:1514] Processing DeleteTablet for tablet e8477d3f66b940538bb74f2cf87a3d47 with delete_type TABLET_DATA_TOMBSTONED (TS 2305b26386ac446da2436463b536d296 not found in new config with opid_index 1234) from {username='slave'} at 127.0.0.1:49878
I20240501 20:11:09.679641 26755 tablet_replica.cc:330] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: stopping tablet replica
I20240501 20:11:09.679883 26755 raft_consensus.cc:2229] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:09.680143 26755 raft_consensus.cc:2258] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:09.681484 26755 ts_tablet_manager.cc:1875] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:09.685662 26755 ts_tablet_manager.cc:1888] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1233
I20240501 20:11:09.685773 26755 log.cc:1196] T e8477d3f66b940538bb74f2cf87a3d47 P 2305b26386ac446da2436463b536d296: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/e8477d3f66b940538bb74f2cf87a3d47
I20240501 20:11:09.686539 20164 catalog_manager.cc:4830] TS 2305b26386ac446da2436463b536d296 (127.14.106.195:35301): tablet e8477d3f66b940538bb74f2cf87a3d47 (table rebalance_test_table_2 [id=5a896728891146bbb1fbcb6a20e31438]) successfully deleted
I20240501 20:11:10.015388 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.049913 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.054484 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:10.095460 20560 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:10.104462 25584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:10.111306 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.124363 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:10.152880 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.308120 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.308895 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.327898 20560 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:10.333721 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.376035 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.380900 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:10.396515 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:10.409511 25584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:10.784873 21872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.791908 22002 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.794462 20560 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:10.822269 20431 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:10.835155 21608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.848727 21740 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:10.851889 20298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:10.858289 25584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
a92b9fd7e9164dc18d7bb7cea432762a | 127.14.106.254:36189 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.106.212:45469 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
00e1df10b53f4936b747b168af137e4b | 127.14.106.199:39669 | HEALTHY | <none> | 1 | 0
133d5e0cbb234be1a5752a784d6d934a | 127.14.106.198:43715 | HEALTHY | <none> | 1 | 0
2305b26386ac446da2436463b536d296 | 127.14.106.195:35301 | HEALTHY | <none> | 3 | 0
28b7e229ab3d4f97a09ca51ac037b008 | 127.14.106.194:39937 | HEALTHY | <none> | 4 | 0
4f8d9458da0b406db99095564f9cf2ed | 127.14.106.193:36237 | HEALTHY | <none> | 1 | 0
79e9c50adfeb44929f5d29524169c6c7 | 127.14.106.196:36027 | HEALTHY | <none> | 1 | 0
a7396c6be769436abac350f7d6e86d55 | 127.14.106.197:35605 | HEALTHY | <none> | 2 | 0
cab748d6deeb4b9699877c77358fe755 | 127.14.106.200:46143 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.14.106.193 | experimental | 127.14.106.193:36237
local_ip_for_outbound_sockets | 127.14.106.194 | experimental | 127.14.106.194:39937
local_ip_for_outbound_sockets | 127.14.106.195 | experimental | 127.14.106.195:35301
local_ip_for_outbound_sockets | 127.14.106.196 | experimental | 127.14.106.196:36027
local_ip_for_outbound_sockets | 127.14.106.197 | experimental | 127.14.106.197:35605
local_ip_for_outbound_sockets | 127.14.106.198 | experimental | 127.14.106.198:43715
local_ip_for_outbound_sockets | 127.14.106.199 | experimental | 127.14.106.199:39669
local_ip_for_outbound_sockets | 127.14.106.200 | experimental | 127.14.106.200:46143
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.106.193:36237
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.106.194:39937
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.106.195:35301
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.106.196:36027
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.14.106.197:35605
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.14.106.198:43715
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.14.106.199:39669
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.14.106.200:46143
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.106.212:45469 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:11:10.929381 14763 log_verifier.cc:126] Checking tablet 01bf239579d84c07adff90ca8f87824d
I20240501 20:11:11.408066 14763 log_verifier.cc:177] Verified matching terms for 1325 ops in tablet 01bf239579d84c07adff90ca8f87824d
I20240501 20:11:11.409598 14763 log_verifier.cc:126] Checking tablet 2e414a3678e646c6b5c6c0dad4fac30e
I20240501 20:11:11.786051 14763 log_verifier.cc:177] Verified matching terms for 1181 ops in tablet 2e414a3678e646c6b5c6c0dad4fac30e
I20240501 20:11:11.787400 14763 log_verifier.cc:126] Checking tablet 4c862c7f6ec24fe2916c97cc7c72401b
I20240501 20:11:12.223982 14763 log_verifier.cc:177] Verified matching terms for 1276 ops in tablet 4c862c7f6ec24fe2916c97cc7c72401b
I20240501 20:11:12.225237 14763 log_verifier.cc:126] Checking tablet 5a45309550b449c1bf0ba233c24140ed
I20240501 20:11:12.553254 14763 log_verifier.cc:177] Verified matching terms for 1114 ops in tablet 5a45309550b449c1bf0ba233c24140ed
I20240501 20:11:12.554491 14763 log_verifier.cc:126] Checking tablet 62a6957734c44303929840f208f5d1d3
I20240501 20:11:12.988487 14763 log_verifier.cc:177] Verified matching terms for 1309 ops in tablet 62a6957734c44303929840f208f5d1d3
I20240501 20:11:12.989814 14763 log_verifier.cc:126] Checking tablet 72363ecf8c08466b8b7f5f4a2d07e1d4
I20240501 20:11:13.352783 14763 log_verifier.cc:177] Verified matching terms for 1230 ops in tablet 72363ecf8c08466b8b7f5f4a2d07e1d4
I20240501 20:11:13.354099 14763 log_verifier.cc:126] Checking tablet 7ff0eebf61824be1aff2cf5c43f45d78
I20240501 20:11:13.779822 14763 log_verifier.cc:177] Verified matching terms for 1350 ops in tablet 7ff0eebf61824be1aff2cf5c43f45d78
I20240501 20:11:13.781214 14763 log_verifier.cc:126] Checking tablet 86d6707287cc48169ae7a548c1c01597
I20240501 20:11:14.141305 14763 log_verifier.cc:177] Verified matching terms for 1236 ops in tablet 86d6707287cc48169ae7a548c1c01597
I20240501 20:11:14.142583 14763 log_verifier.cc:126] Checking tablet afabe33672c647059a9c125c76906fac
I20240501 20:11:14.552609 14763 log_verifier.cc:177] Verified matching terms for 1301 ops in tablet afabe33672c647059a9c125c76906fac
I20240501 20:11:14.553961 14763 log_verifier.cc:126] Checking tablet b9ccf34e292a480aa7283631edb5a0d2
I20240501 20:11:14.927181 14763 log_verifier.cc:177] Verified matching terms for 1304 ops in tablet b9ccf34e292a480aa7283631edb5a0d2
I20240501 20:11:14.928680 14763 log_verifier.cc:126] Checking tablet e4c119cb173942819ec01f7f3797c9d7
I20240501 20:11:15.329365 14763 log_verifier.cc:177] Verified matching terms for 1253 ops in tablet e4c119cb173942819ec01f7f3797c9d7
I20240501 20:11:15.330718 14763 log_verifier.cc:126] Checking tablet e4dff67082a6473f994c09821a0d7a3d
I20240501 20:11:15.341146 14763 log_verifier.cc:177] Verified matching terms for 4 ops in tablet e4dff67082a6473f994c09821a0d7a3d
I20240501 20:11:15.341230 14763 log_verifier.cc:126] Checking tablet e74fb1e0e017408f946a8df8ae304f13
I20240501 20:11:15.724478 14763 log_verifier.cc:177] Verified matching terms for 1194 ops in tablet e74fb1e0e017408f946a8df8ae304f13
I20240501 20:11:15.725695 14763 log_verifier.cc:126] Checking tablet e8477d3f66b940538bb74f2cf87a3d47
I20240501 20:11:16.091856 14763 log_verifier.cc:177] Verified matching terms for 1234 ops in tablet e8477d3f66b940538bb74f2cf87a3d47
I20240501 20:11:16.093111 14763 log_verifier.cc:126] Checking tablet ee446411ba4e4da2b295080da28d40b9
I20240501 20:11:16.476158 14763 log_verifier.cc:177] Verified matching terms for 1198 ops in tablet ee446411ba4e4da2b295080da28d40b9
I20240501 20:11:16.477347 14763 log_verifier.cc:126] Checking tablet f9480642e71d46679005d4f55d2e88f8
I20240501 20:11:16.889389 14763 log_verifier.cc:177] Verified matching terms for 1410 ops in tablet f9480642e71d46679005d4f55d2e88f8
I20240501 20:11:16.897245 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 20238
I20240501 20:11:16.921422 20359 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:17.237805 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 20238
W20240501 20:11:17.287977 21692 connection.cc:535] client connection to 127.14.106.193:36237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:17.288283 21692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:17.288400 20166 connection.cc:535] server connection from 127.14.106.193:42575 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:17.289100 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 20369
W20240501 20:11:17.290405 21692 consensus_peers.cc:485] T afabe33672c647059a9c125c76906fac P a7396c6be769436abac350f7d6e86d55 -> Peer 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237): Couldn't send request to peer 4f8d9458da0b406db99095564f9cf2ed. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:36237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.295091 20491 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:11:17.295084 25537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:36237: connect: Connection refused (error 111)
W20240501 20:11:17.296741 25537 consensus_peers.cc:485] T 5a45309550b449c1bf0ba233c24140ed P cab748d6deeb4b9699877c77358fe755 -> Peer 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237): Couldn't send request to peer 4f8d9458da0b406db99095564f9cf2ed. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:36237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.614275 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 20369
W20240501 20:11:17.664805 21827 connection.cc:535] server connection from 127.14.106.194:58963 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:17.665294 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 20499
I20240501 20:11:17.671000 20621 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:17.991586 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 20499
I20240501 20:11:18.042698 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 21546
W20240501 20:11:18.042598 21958 connection.cc:535] server connection from 127.14.106.195:52663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:18.042781 21561 connection.cc:535] server connection from 127.14.106.195:52347 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:18.049218 21668 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:18.319025 26919 raft_consensus.cc:482] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4f8d9458da0b406db99095564f9cf2ed)
I20240501 20:11:18.319238 26919 raft_consensus.cc:504] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "4f8d9458da0b406db99095564f9cf2ed" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 36237 } } peers { permanent_uuid: "79e9c50adfeb44929f5d29524169c6c7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 36027 } }
I20240501 20:11:18.319972 26919 leader_election.cc:288] T 7ff0eebf61824be1aff2cf5c43f45d78 P 79e9c50adfeb44929f5d29524169c6c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f8d9458da0b406db99095564f9cf2ed (127.14.106.193:36237)
I20240501 20:11:18.326745 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 21546
I20240501 20:11:18.373178 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 21677
I20240501 20:11:18.378585 21802 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:11:18.404601 21826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.195:35301: connect: Connection refused (error 111)
W20240501 20:11:18.406158 21826 consensus_peers.cc:485] T b9ccf34e292a480aa7283631edb5a0d2 P 133d5e0cbb234be1a5752a784d6d934a -> Peer 2305b26386ac446da2436463b536d296 (127.14.106.195:35301): Couldn't send request to peer 2305b26386ac446da2436463b536d296. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.195:35301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.633739 26920 raft_consensus.cc:482] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28b7e229ab3d4f97a09ca51ac037b008)
I20240501 20:11:18.633975 26920 raft_consensus.cc:504] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "133d5e0cbb234be1a5752a784d6d934a" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 43715 } }
I20240501 20:11:18.634727 26920 leader_election.cc:288] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
W20240501 20:11:18.636099 21826 leader_election.cc:334] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): Network error: Client connection negotiation failed: client connection to 127.14.106.194:39937: connect: Connection refused (error 111)
I20240501 20:11:18.636238 21826 leader_election.cc:302] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 133d5e0cbb234be1a5752a784d6d934a; no voters: 28b7e229ab3d4f97a09ca51ac037b008
I20240501 20:11:18.636536 26920 raft_consensus.cc:2726] T 01bf239579d84c07adff90ca8f87824d P 133d5e0cbb234be1a5752a784d6d934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:18.657764 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 21677
W20240501 20:11:18.705399 20163 connection.cc:535] server connection from 127.14.106.197:38887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:18.705487 21955 connection.cc:535] server connection from 127.14.106.197:35253 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:18.705734 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 21810
I20240501 20:11:18.711061 21932 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:18.776368 26941 raft_consensus.cc:482] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28b7e229ab3d4f97a09ca51ac037b008)
I20240501 20:11:18.776573 26941 raft_consensus.cc:504] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "28b7e229ab3d4f97a09ca51ac037b008" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 39937 } } peers { permanent_uuid: "00e1df10b53f4936b747b168af137e4b" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 39669 } }
I20240501 20:11:18.777235 26941 leader_election.cc:288] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937)
W20240501 20:11:18.778205 21956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.194:39937: connect: Connection refused (error 111)
W20240501 20:11:18.779458 21956 leader_election.cc:334] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28b7e229ab3d4f97a09ca51ac037b008 (127.14.106.194:39937): Network error: Client connection negotiation failed: client connection to 127.14.106.194:39937: connect: Connection refused (error 111)
I20240501 20:11:18.779567 21956 leader_election.cc:302] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 00e1df10b53f4936b747b168af137e4b; no voters: 28b7e229ab3d4f97a09ca51ac037b008
I20240501 20:11:18.779812 26941 raft_consensus.cc:2726] T ee446411ba4e4da2b295080da28d40b9 P 00e1df10b53f4936b747b168af137e4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:11:18.814473 25540 consensus_peers.cc:485] T 72363ecf8c08466b8b7f5f4a2d07e1d4 P cab748d6deeb4b9699877c77358fe755 -> Peer a7396c6be769436abac350f7d6e86d55 (127.14.106.197:35605): Couldn't send request to peer a7396c6be769436abac350f7d6e86d55. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:35605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.984670 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 21810
I20240501 20:11:19.032723 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 21940
I20240501 20:11:19.038234 22062 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:19.302888 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 21940
I20240501 20:11:19.348428 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 25522
I20240501 20:11:19.354351 25644 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:19.565644 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 25522
I20240501 20:11:19.597842 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 20148
I20240501 20:11:19.598822 20209 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:19.844969 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 20148
2024-05-01T20:11:19Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (117871 ms)
[----------] 1 test from RebalanceParamTest (117871 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240501 20:11:20.000869 14763 test_util.cc:268] Using random seed: 385764696
I20240501 20:11:20.002498 14763 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:11:20.002564 14763 ts_itest-base.cc:113] --------------
I20240501 20:11:20.002601 14763 ts_itest-base.cc:114] 8 tablet servers
I20240501 20:11:20.002671 14763 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:11:20.002724 14763 ts_itest-base.cc:116] --------------
2024-05-01T20:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:11:20Z Disabled control of system clock
I20240501 20:11:20.057436 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:34283
--webserver_interface=127.14.106.254
--webserver_port=0
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.106.254:34283
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 20:11:20.405149 26963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:20.405437 26963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:20.405573 26963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:20.412910 26963 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:20.413039 26963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:11:20.413126 26963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:20.413211 26963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:11:20.413278 26963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:11:20.422176 26963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.106.254:34283
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:20.423494 26963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:20.425208 26963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:20.435778 26968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:20.436250 26971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:20.436695 26969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:20.437355 26963 server_base.cc:1034] running on GCE node
I20240501 20:11:20.438040 26963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:20.439359 26963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:20.440572 26963 hybrid_clock.cc:648] HybridClock initialized: now 1714594280440529 us; error 36 us; skew 500 ppm
I20240501 20:11:20.440893 26963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:20.443320 26963 webserver.cc:460] Webserver started at http://127.14.106.254:35851/ using document root <none> and password file <none>
I20240501 20:11:20.444072 26963 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:20.444197 26963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:20.444525 26963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:20.447175 26963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "423345d7db4f4b2eb388943cdadfc07a"
format_stamp: "Formatted at 2024-05-01 20:11:20 on dist-test-slave-pg4x"
I20240501 20:11:20.447955 26963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "423345d7db4f4b2eb388943cdadfc07a"
format_stamp: "Formatted at 2024-05-01 20:11:20 on dist-test-slave-pg4x"
I20240501 20:11:20.453121 26963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:11:20.456435 26977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:20.457700 26963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:20.457864 26963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
uuid: "423345d7db4f4b2eb388943cdadfc07a"
format_stamp: "Formatted at 2024-05-01 20:11:20 on dist-test-slave-pg4x"
I20240501 20:11:20.458076 26963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:20.476255 26963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:20.477032 26963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:20.477309 26963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:20.495123 26963 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.254:34283
I20240501 20:11:20.495146 27028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.254:34283 every 8 connection(s)
I20240501 20:11:20.496479 26963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:11:20.500056 27029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:20.506892 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 26963
I20240501 20:11:20.507107 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:11:20.509565 27029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a: Bootstrap starting.
I20240501 20:11:20.512686 27029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:20.514017 27029 log.cc:824] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:20.516829 27029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a: No bootstrap required, opened a new log
I20240501 20:11:20.523801 27029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423345d7db4f4b2eb388943cdadfc07a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34283 } }
I20240501 20:11:20.524255 27029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:20.524346 27029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 423345d7db4f4b2eb388943cdadfc07a, State: Initialized, Role: FOLLOWER
I20240501 20:11:20.525018 27029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423345d7db4f4b2eb388943cdadfc07a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34283 } }
I20240501 20:11:20.525224 27029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:20.525362 27029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:20.525542 27029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:20.527645 27029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423345d7db4f4b2eb388943cdadfc07a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34283 } }
I20240501 20:11:20.528072 27029 leader_election.cc:302] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 423345d7db4f4b2eb388943cdadfc07a; no voters:
I20240501 20:11:20.528584 27029 leader_election.cc:288] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:20.528731 27034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:20.529408 27034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [term 1 LEADER]: Becoming Leader. State: Replica: 423345d7db4f4b2eb388943cdadfc07a, State: Running, Role: LEADER
I20240501 20:11:20.530077 27034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423345d7db4f4b2eb388943cdadfc07a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34283 } }
I20240501 20:11:20.530663 27029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:11:20.532995 27036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 423345d7db4f4b2eb388943cdadfc07a. Latest consensus state: current_term: 1 leader_uuid: "423345d7db4f4b2eb388943cdadfc07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423345d7db4f4b2eb388943cdadfc07a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34283 } } }
I20240501 20:11:20.533423 27036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:20.533336 27035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "423345d7db4f4b2eb388943cdadfc07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423345d7db4f4b2eb388943cdadfc07a" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 34283 } } }
I20240501 20:11:20.533617 27035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:20.535102 27043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:11:20.540122 27043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:11:20.548069 27043 catalog_manager.cc:1347] Generated new cluster ID: a1753b5d38ca4f24ab1c622be95a5bce
I20240501 20:11:20.548185 27043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:11:20.596304 27043 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:11:20.597580 27043 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:11:20.606659 27043 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 423345d7db4f4b2eb388943cdadfc07a: Generated new TSK 0
I20240501 20:11:20.607481 27043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:11:20.621376 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.193:0
--local_ip_for_outbound_sockets=127.14.106.193
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:11:20.992458 27053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:20.992735 27053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:20.992825 27053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:20.999990 27053 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:21.000119 27053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:21.000237 27053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.193
I20240501 20:11:21.009034 27053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.193
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:21.010291 27053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:21.011971 27053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:21.023945 27058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:21.024330 27059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:21.024724 27061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:21.025310 27053 server_base.cc:1034] running on GCE node
I20240501 20:11:21.025908 27053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:21.026854 27053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:21.028074 27053 hybrid_clock.cc:648] HybridClock initialized: now 1714594281028009 us; error 52 us; skew 500 ppm
I20240501 20:11:21.028389 27053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:21.030689 27053 webserver.cc:460] Webserver started at http://127.14.106.193:34513/ using document root <none> and password file <none>
I20240501 20:11:21.031443 27053 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:21.031575 27053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:21.031908 27053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:21.034551 27053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4d603ea5d17441b3bac562bf08ec83fe"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.035332 27053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4d603ea5d17441b3bac562bf08ec83fe"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.040422 27053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:11:21.043780 27067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:21.045075 27053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:11:21.045240 27053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4d603ea5d17441b3bac562bf08ec83fe"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.045475 27053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:21.070133 27053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:21.071090 27053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:21.071369 27053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:21.072314 27053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:21.073769 27053 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:21.073885 27053 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:21.073998 27053 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:21.074071 27053 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:21.104369 27053 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.193:46293
I20240501 20:11:21.104435 27179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.193:46293 every 8 connection(s)
I20240501 20:11:21.105697 27053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:11:21.111431 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 27053
I20240501 20:11:21.111660 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:11:21.114734 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.194:0
--local_ip_for_outbound_sockets=127.14.106.194
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:21.118347 27180 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:21.118690 27180 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:21.119553 27180 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:21.121562 26994 ts_manager.cc:194] Registered new tserver with Master: 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:21.123061 26994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.193:39129
W20240501 20:11:21.449234 27184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:21.449520 27184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:21.449666 27184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:21.457108 27184 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:21.457237 27184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:21.457402 27184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.194
I20240501 20:11:21.466392 27184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.194
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:21.467696 27184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:21.469354 27184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:21.481045 27189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:21.481551 27190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:21.482060 27192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:21.482671 27184 server_base.cc:1034] running on GCE node
I20240501 20:11:21.483191 27184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:21.484081 27184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:21.485277 27184 hybrid_clock.cc:648] HybridClock initialized: now 1714594281485223 us; error 39 us; skew 500 ppm
I20240501 20:11:21.485597 27184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:21.487856 27184 webserver.cc:460] Webserver started at http://127.14.106.194:45765/ using document root <none> and password file <none>
I20240501 20:11:21.488562 27184 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:21.488672 27184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:21.488977 27184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:21.491565 27184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d8b307156e1346bd9eecf655f49924c7"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.492280 27184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d8b307156e1346bd9eecf655f49924c7"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.497627 27184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:11:21.501046 27198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:21.502492 27184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:21.502682 27184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d8b307156e1346bd9eecf655f49924c7"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.502871 27184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:21.521293 27184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:21.522107 27184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:21.522370 27184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:21.523320 27184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:21.524684 27184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:21.524780 27184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:21.524891 27184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:21.524964 27184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:21.557001 27184 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.194:46281
I20240501 20:11:21.557055 27310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.194:46281 every 8 connection(s)
I20240501 20:11:21.558282 27184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:11:21.567836 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 27184
I20240501 20:11:21.568034 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:11:21.570717 27311 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:21.571082 27311 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:21.571619 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.195:0
--local_ip_for_outbound_sockets=127.14.106.195
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:21.572074 27311 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:21.573628 26993 ts_manager.cc:194] Registered new tserver with Master: d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:21.574522 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.194:51605
W20240501 20:11:21.905630 27314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:21.905907 27314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:21.906005 27314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:21.913556 27314 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:21.913686 27314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:21.913805 27314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.195
I20240501 20:11:21.922821 27314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.195
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:21.924111 27314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:21.925782 27314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:21.937788 27319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:21.938222 27322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:21.938599 27320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:21.939224 27314 server_base.cc:1034] running on GCE node
I20240501 20:11:21.939764 27314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:21.940639 27314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:21.941836 27314 hybrid_clock.cc:648] HybridClock initialized: now 1714594281941794 us; error 43 us; skew 500 ppm
I20240501 20:11:21.942150 27314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:21.948818 27314 webserver.cc:460] Webserver started at http://127.14.106.195:40833/ using document root <none> and password file <none>
I20240501 20:11:21.949568 27314 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:21.949683 27314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:21.950001 27314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:21.952553 27314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e677127814314770b3370764d0ff9d06"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.953281 27314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e677127814314770b3370764d0ff9d06"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.958140 27314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:11:21.961338 27328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:21.962586 27314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:21.962786 27314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e677127814314770b3370764d0ff9d06"
format_stamp: "Formatted at 2024-05-01 20:11:21 on dist-test-slave-pg4x"
I20240501 20:11:21.962992 27314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:21.979779 27314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:21.980594 27314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:21.980836 27314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:21.981737 27314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:21.983374 27314 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:21.983464 27314 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:21.983549 27314 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:21.983604 27314 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:22.015609 27314 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.195:35443
I20240501 20:11:22.015666 27440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.195:35443 every 8 connection(s)
I20240501 20:11:22.016943 27314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:22.024190 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 27314
I20240501 20:11:22.024412 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:11:22.027902 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:0
--local_ip_for_outbound_sockets=127.14.106.196
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:22.029763 27441 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:22.030030 27441 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:22.030802 27441 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:22.032171 26993 ts_manager.cc:194] Registered new tserver with Master: e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:22.033226 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.195:46823
I20240501 20:11:22.126147 27180 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
W20240501 20:11:22.366596 27444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:22.367022 27444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:22.367172 27444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:22.374907 27444 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:22.375068 27444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:22.375236 27444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:11:22.384135 27444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:22.385387 27444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:22.387084 27444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:22.398669 27452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:22.399093 27450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:22.399542 27449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:22.400099 27444 server_base.cc:1034] running on GCE node
I20240501 20:11:22.400689 27444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:22.401664 27444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:22.402851 27444 hybrid_clock.cc:648] HybridClock initialized: now 1714594282402791 us; error 54 us; skew 500 ppm
I20240501 20:11:22.403147 27444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:22.405339 27444 webserver.cc:460] Webserver started at http://127.14.106.196:42333/ using document root <none> and password file <none>
I20240501 20:11:22.406044 27444 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:22.406163 27444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:22.406499 27444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:22.409085 27444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "524f5e87a3ce40378bde76e667a4ece7"
format_stamp: "Formatted at 2024-05-01 20:11:22 on dist-test-slave-pg4x"
I20240501 20:11:22.409819 27444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "524f5e87a3ce40378bde76e667a4ece7"
format_stamp: "Formatted at 2024-05-01 20:11:22 on dist-test-slave-pg4x"
I20240501 20:11:22.414867 27444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 20:11:22.418064 27458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:22.419421 27444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:22.419584 27444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "524f5e87a3ce40378bde76e667a4ece7"
format_stamp: "Formatted at 2024-05-01 20:11:22 on dist-test-slave-pg4x"
I20240501 20:11:22.419768 27444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:22.442459 27444 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:22.443348 27444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:22.443631 27444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:22.444566 27444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:22.446031 27444 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:22.446128 27444 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:22.446241 27444 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:22.446318 27444 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:22.478253 27444 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:43861
I20240501 20:11:22.478312 27570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:43861 every 8 connection(s)
I20240501 20:11:22.479508 27444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:22.489776 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 27444
I20240501 20:11:22.489990 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:11:22.491048 27571 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:22.491341 27571 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:22.492110 27571 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:22.493167 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:0
--local_ip_for_outbound_sockets=127.14.106.197
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:22.493552 26993 ts_manager.cc:194] Registered new tserver with Master: 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:22.494663 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:47021
I20240501 20:11:22.577529 27311 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
W20240501 20:11:22.834388 27574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:22.834695 27574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:22.834837 27574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:22.842327 27574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:22.842475 27574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:22.842661 27574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:11:22.851680 27574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:22.852989 27574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:22.854682 27574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:22.866353 27579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:22.866813 27582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:22.867172 27580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:22.867790 27574 server_base.cc:1034] running on GCE node
I20240501 20:11:22.868342 27574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:22.869221 27574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:22.870437 27574 hybrid_clock.cc:648] HybridClock initialized: now 1714594282870386 us; error 46 us; skew 500 ppm
I20240501 20:11:22.870795 27574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:22.873030 27574 webserver.cc:460] Webserver started at http://127.14.106.197:37495/ using document root <none> and password file <none>
I20240501 20:11:22.873742 27574 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:22.873858 27574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:22.874166 27574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:22.876757 27574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "565cade0058540a4aa18d7c7a25affbe"
format_stamp: "Formatted at 2024-05-01 20:11:22 on dist-test-slave-pg4x"
I20240501 20:11:22.877475 27574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "565cade0058540a4aa18d7c7a25affbe"
format_stamp: "Formatted at 2024-05-01 20:11:22 on dist-test-slave-pg4x"
I20240501 20:11:22.882328 27574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:11:22.885511 27588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:22.886714 27574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:11:22.886864 27574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "565cade0058540a4aa18d7c7a25affbe"
format_stamp: "Formatted at 2024-05-01 20:11:22 on dist-test-slave-pg4x"
I20240501 20:11:22.887043 27574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:22.906176 27574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:22.907042 27574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:22.907311 27574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:22.908324 27574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:22.909770 27574 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:22.909869 27574 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:22.909986 27574 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:22.910068 27574 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:22.952998 27574 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:40723
I20240501 20:11:22.953048 27700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:40723 every 8 connection(s)
I20240501 20:11:22.954347 27574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:11:22.956965 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 27574
I20240501 20:11:22.957165 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:11:22.960150 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.198:0
--local_ip_for_outbound_sockets=127.14.106.198
--webserver_interface=127.14.106.198
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:22.967340 27701 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:22.967671 27701 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:22.968605 27701 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:22.970070 26993 ts_manager.cc:194] Registered new tserver with Master: 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
I20240501 20:11:22.970896 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:46343
I20240501 20:11:23.035687 27441 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
W20240501 20:11:23.301962 27704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:23.302225 27704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:23.302371 27704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:23.309638 27704 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:23.309748 27704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:23.309913 27704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.198
I20240501 20:11:23.318799 27704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.106.198
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.198
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:23.320050 27704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:23.321708 27704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:23.333850 27709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:23.334312 27712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:23.334867 27710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:23.335403 27704 server_base.cc:1034] running on GCE node
I20240501 20:11:23.336004 27704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:23.336864 27704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:23.338090 27704 hybrid_clock.cc:648] HybridClock initialized: now 1714594283338043 us; error 42 us; skew 500 ppm
I20240501 20:11:23.338387 27704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:23.340633 27704 webserver.cc:460] Webserver started at http://127.14.106.198:33949/ using document root <none> and password file <none>
I20240501 20:11:23.341339 27704 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:23.341449 27704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:23.341760 27704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:23.344265 27704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "d56cbcb06b9e418cb617de6a34b946b0"
format_stamp: "Formatted at 2024-05-01 20:11:23 on dist-test-slave-pg4x"
I20240501 20:11:23.344975 27704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "d56cbcb06b9e418cb617de6a34b946b0"
format_stamp: "Formatted at 2024-05-01 20:11:23 on dist-test-slave-pg4x"
I20240501 20:11:23.350184 27704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240501 20:11:23.353523 27718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:23.354894 27704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:23.355036 27704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d56cbcb06b9e418cb617de6a34b946b0"
format_stamp: "Formatted at 2024-05-01 20:11:23 on dist-test-slave-pg4x"
I20240501 20:11:23.355202 27704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:23.368041 27704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:23.368734 27704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:23.368969 27704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:23.369789 27704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:23.371142 27704 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:23.371232 27704 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:23.371309 27704 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:23.371367 27704 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:23.403616 27704 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.198:46843
I20240501 20:11:23.403668 27830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.198:46843 every 8 connection(s)
I20240501 20:11:23.404886 27704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:11:23.413833 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 27704
I20240501 20:11:23.414036 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:11:23.416929 27831 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:23.417200 27831 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:23.417186 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.199:0
--local_ip_for_outbound_sockets=127.14.106.199
--webserver_interface=127.14.106.199
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:23.417943 27831 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:23.419212 26993 ts_manager.cc:194] Registered new tserver with Master: d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843)
I20240501 20:11:23.419986 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.198:50097
I20240501 20:11:23.497452 27571 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
W20240501 20:11:23.765640 27834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:23.765935 27834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:23.766063 27834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:23.773676 27834 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:23.773799 27834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:23.773973 27834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.199
I20240501 20:11:23.782897 27834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.106.199
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.199
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:23.784219 27834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:23.785988 27834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:23.797899 27839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:23.798429 27842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:23.798939 27840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:23.799492 27834 server_base.cc:1034] running on GCE node
I20240501 20:11:23.800112 27834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:23.801057 27834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:23.802255 27834 hybrid_clock.cc:648] HybridClock initialized: now 1714594283802209 us; error 41 us; skew 500 ppm
I20240501 20:11:23.802570 27834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:23.804773 27834 webserver.cc:460] Webserver started at http://127.14.106.199:35385/ using document root <none> and password file <none>
I20240501 20:11:23.805481 27834 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:23.805599 27834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:23.805918 27834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:23.808614 27834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "212de34e3aa24fa1be12c7421df985fc"
format_stamp: "Formatted at 2024-05-01 20:11:23 on dist-test-slave-pg4x"
I20240501 20:11:23.809322 27834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "212de34e3aa24fa1be12c7421df985fc"
format_stamp: "Formatted at 2024-05-01 20:11:23 on dist-test-slave-pg4x"
I20240501 20:11:23.814219 27834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 20:11:23.817516 27848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:23.818888 27834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:23.819057 27834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "212de34e3aa24fa1be12c7421df985fc"
format_stamp: "Formatted at 2024-05-01 20:11:23 on dist-test-slave-pg4x"
I20240501 20:11:23.819245 27834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:23.856709 27834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:23.857559 27834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:23.857836 27834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:23.858842 27834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:23.860325 27834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:23.860414 27834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:23.860527 27834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:23.860596 27834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:23.893164 27834 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.199:32969
I20240501 20:11:23.893234 27960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.199:32969 every 8 connection(s)
I20240501 20:11:23.894462 27834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:11:23.899920 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 27834
I20240501 20:11:23.900118 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:11:23.903429 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.200:0
--local_ip_for_outbound_sockets=127.14.106.200
--webserver_interface=127.14.106.200
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:23.908824 27961 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:23.909114 27961 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:23.909862 27961 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:23.911206 26993 ts_manager.cc:194] Registered new tserver with Master: 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
I20240501 20:11:23.912005 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.199:47119
I20240501 20:11:23.973611 27701 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
W20240501 20:11:24.243441 27964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:24.243705 27964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:24.243808 27964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:24.251197 27964 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:24.251319 27964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:24.251437 27964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.200
I20240501 20:11:24.260409 27964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.106.200
--webserver_port=0
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.200
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:24.261626 27964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:24.263355 27964 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:24.275524 27969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:24.275960 27970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:24.276428 27972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:24.277040 27964 server_base.cc:1034] running on GCE node
I20240501 20:11:24.277630 27964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:24.278560 27964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:24.279790 27964 hybrid_clock.cc:648] HybridClock initialized: now 1714594284279744 us; error 38 us; skew 500 ppm
I20240501 20:11:24.280103 27964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:24.282269 27964 webserver.cc:460] Webserver started at http://127.14.106.200:45359/ using document root <none> and password file <none>
I20240501 20:11:24.283027 27964 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:24.283144 27964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:24.283442 27964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:24.285948 27964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e957d29835544ac587917cc067fb7423"
format_stamp: "Formatted at 2024-05-01 20:11:24 on dist-test-slave-pg4x"
I20240501 20:11:24.286690 27964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e957d29835544ac587917cc067fb7423"
format_stamp: "Formatted at 2024-05-01 20:11:24 on dist-test-slave-pg4x"
I20240501 20:11:24.291666 27964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240501 20:11:24.294826 27978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:24.296202 27964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:11:24.296365 27964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e957d29835544ac587917cc067fb7423"
format_stamp: "Formatted at 2024-05-01 20:11:24 on dist-test-slave-pg4x"
I20240501 20:11:24.296556 27964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:24.312649 27964 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:24.313397 27964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:24.313652 27964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:24.314522 27964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:24.315917 27964 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:24.316037 27964 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:24.316133 27964 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:24.316190 27964 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:24.347615 27964 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.200:44949
I20240501 20:11:24.347679 28090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.200:44949 every 8 connection(s)
I20240501 20:11:24.348888 27964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:11:24.356823 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 27964
I20240501 20:11:24.357048 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:11:24.361436 28091 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:24.361706 28091 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:24.362447 28091 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:24.363711 26993 ts_manager.cc:194] Registered new tserver with Master: e957d29835544ac587917cc067fb7423 (127.14.106.200:44949)
I20240501 20:11:24.364467 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.200:58803
I20240501 20:11:24.373059 14763 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:11:24.389256 14763 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:11:24.389400 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27574
I20240501 20:11:24.397516 27696 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:24.422256 27831 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
I20240501 20:11:24.479832 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 27574
I20240501 20:11:24.496845 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27704
I20240501 20:11:24.505759 27826 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:24.586122 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 27704
I20240501 20:11:24.602828 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27834
I20240501 20:11:24.610339 27956 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:24.689785 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 27834
I20240501 20:11:24.706187 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27964
I20240501 20:11:24.714120 28086 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:24.794587 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 27964
I20240501 20:11:28.563102 26993 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:28.620664 27506 tablet_service.cc:1467] Processing CreateTablet for tablet 26cb2329d176441ca9127187db2d157a (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.621522 27505 tablet_service.cc:1467] Processing CreateTablet for tablet f02a619bc1584d53ac902c8d8baff976 (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.621235 27246 tablet_service.cc:1467] Processing CreateTablet for tablet 50bf9058852a4ea49733326057d5ff41 (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.622704 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bf9058852a4ea49733326057d5ff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.623071 27245 tablet_service.cc:1467] Processing CreateTablet for tablet 26cb2329d176441ca9127187db2d157a (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.623574 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cb2329d176441ca9127187db2d157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.623642 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02a619bc1584d53ac902c8d8baff976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.624605 27506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cb2329d176441ca9127187db2d157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.625238 27376 tablet_service.cc:1467] Processing CreateTablet for tablet 50bf9058852a4ea49733326057d5ff41 (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.626595 27376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bf9058852a4ea49733326057d5ff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.629669 27115 tablet_service.cc:1467] Processing CreateTablet for tablet 50bf9058852a4ea49733326057d5ff41 (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.625217 27375 tablet_service.cc:1467] Processing CreateTablet for tablet f02a619bc1584d53ac902c8d8baff976 (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.630606 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02a619bc1584d53ac902c8d8baff976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.631008 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bf9058852a4ea49733326057d5ff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.643604 27114 tablet_service.cc:1467] Processing CreateTablet for tablet 26cb2329d176441ca9127187db2d157a (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.643846 27113 tablet_service.cc:1467] Processing CreateTablet for tablet f02a619bc1584d53ac902c8d8baff976 (DEFAULT_TABLE table=rebalance_test_table_0 [id=565e816257734920a19d1af631224041]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:28.644214 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cb2329d176441ca9127187db2d157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.644517 27113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02a619bc1584d53ac902c8d8baff976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.660630 28121 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:28.662747 28123 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:28.665125 28121 tablet_bootstrap.cc:654] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.667515 28121 log.cc:824] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:28.668126 28120 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:28.668769 28122 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:28.669375 28123 tablet_bootstrap.cc:654] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.670261 28123 log.cc:824] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:28.671383 28120 tablet_bootstrap.cc:654] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.671957 28122 tablet_bootstrap.cc:654] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.672577 28120 log.cc:824] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:28.673110 28122 log.cc:824] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:28.673233 28121 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:28.673282 28123 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:28.673520 28121 ts_tablet_manager.cc:1367] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20240501 20:11:28.673563 28123 ts_tablet_manager.cc:1367] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240501 20:11:28.679299 28122 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:28.679651 28122 ts_tablet_manager.cc:1367] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 20:11:28.682690 28121 raft_consensus.cc:348] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.682688 28123 raft_consensus.cc:348] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.683126 28121 raft_consensus.cc:374] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.683235 28121 raft_consensus.cc:729] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.683282 28123 raft_consensus.cc:374] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.683477 28123 raft_consensus.cc:729] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.683954 28121 consensus_queue.cc:260] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.684324 28123 consensus_queue.cc:260] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.688448 28121 ts_tablet_manager.cc:1398] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20240501 20:11:28.688894 28120 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:28.689167 28120 ts_tablet_manager.cc:1367] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20240501 20:11:28.689311 28121 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:28.691922 28121 tablet_bootstrap.cc:654] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.691954 28123 ts_tablet_manager.cc:1398] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.018s user 0.009s sys 0.006s
I20240501 20:11:28.692476 28123 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:28.694672 28123 tablet_bootstrap.cc:654] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.697762 28122 raft_consensus.cc:348] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.698184 28122 raft_consensus.cc:374] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.698294 28122 raft_consensus.cc:729] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.699119 28122 consensus_queue.cc:260] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.701056 28121 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:28.701090 28123 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:28.701213 28123 ts_tablet_manager.cc:1367] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:11:28.701251 28121 ts_tablet_manager.cc:1367] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240501 20:11:28.702271 28121 raft_consensus.cc:348] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.702271 28123 raft_consensus.cc:348] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.702476 28123 raft_consensus.cc:374] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.702585 28121 raft_consensus.cc:374] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.702574 28123 raft_consensus.cc:729] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.702752 28121 raft_consensus.cc:729] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.702837 28123 consensus_queue.cc:260] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.703166 28121 consensus_queue.cc:260] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.703461 28123 ts_tablet_manager.cc:1398] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:28.703891 28121 ts_tablet_manager.cc:1398] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:28.703862 28120 raft_consensus.cc:348] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.704285 28120 raft_consensus.cc:374] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.704382 28120 raft_consensus.cc:729] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.705231 28120 consensus_queue.cc:260] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.708540 28122 ts_tablet_manager.cc:1398] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.029s user 0.016s sys 0.004s
I20240501 20:11:28.709070 28122 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:28.709923 28120 ts_tablet_manager.cc:1398] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.021s user 0.015s sys 0.000s
I20240501 20:11:28.710358 28120 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:28.711845 28122 tablet_bootstrap.cc:654] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.712365 28120 tablet_bootstrap.cc:654] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.714484 28122 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:28.714655 28122 ts_tablet_manager.cc:1367] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:11:28.714933 28120 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:28.715086 28120 ts_tablet_manager.cc:1367] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:11:28.715178 28132 raft_consensus.cc:482] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:28.715415 28132 raft_consensus.cc:504] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.715566 28122 raft_consensus.cc:348] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.715754 28122 raft_consensus.cc:374] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.715860 28122 raft_consensus.cc:729] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.716055 28120 raft_consensus.cc:348] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.716109 28122 consensus_queue.cc:260] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.716248 28120 raft_consensus.cc:374] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.716343 28120 raft_consensus.cc:729] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.716588 28120 consensus_queue.cc:260] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.717195 28120 ts_tablet_manager.cc:1398] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:28.717861 28122 ts_tablet_manager.cc:1398] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:28.718122 28122 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:28.719972 28122 tablet_bootstrap.cc:654] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:28.723641 28132 leader_election.cc:288] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:28.724195 28122 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:28.724352 28122 ts_tablet_manager.cc:1367] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240501 20:11:28.725069 28122 raft_consensus.cc:348] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.725268 28122 raft_consensus.cc:374] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:28.725356 28122 raft_consensus.cc:729] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:28.725603 28122 consensus_queue.cc:260] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.726114 28122 ts_tablet_manager.cc:1398] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 20:11:28.735473 27572 tablet.cc:2369] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:28.737146 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50bf9058852a4ea49733326057d5ff41" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:28.737685 27396 raft_consensus.cc:2454] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:28.737749 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50bf9058852a4ea49733326057d5ff41" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:28.738211 27266 raft_consensus.cc:2454] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:28.738322 27071 leader_election.cc:302] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:28.738762 28132 raft_consensus.cc:2781] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:28.738888 28132 raft_consensus.cc:482] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:28.738983 28132 raft_consensus.cc:3037] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:28.740964 28132 raft_consensus.cc:504] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.741596 28132 leader_election.cc:288] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:28.741987 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50bf9058852a4ea49733326057d5ff41" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:28.742187 27396 raft_consensus.cc:3037] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:28.742424 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50bf9058852a4ea49733326057d5ff41" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:28.742645 27266 raft_consensus.cc:3037] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:28.745266 27396 raft_consensus.cc:2454] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:28.745651 27266 raft_consensus.cc:2454] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:28.745680 27071 leader_election.cc:302] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:28.746021 28132 raft_consensus.cc:2781] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:28.746209 28132 raft_consensus.cc:686] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:28.746681 28132 consensus_queue.cc:237] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:28.751258 26993 catalog_manager.cc:5484] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
W20240501 20:11:28.773439 27442 tablet.cc:2369] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:28.801538 28128 raft_consensus.cc:482] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:28.801739 28128 raft_consensus.cc:504] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.803450 28128 leader_election.cc:288] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:28.811096 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cb2329d176441ca9127187db2d157a" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:28.811138 27526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cb2329d176441ca9127187db2d157a" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:28.811482 27135 raft_consensus.cc:2454] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 0.
I20240501 20:11:28.811580 27526 raft_consensus.cc:2454] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 0.
I20240501 20:11:28.812023 27201 leader_election.cc:302] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:28.812367 28128 raft_consensus.cc:2781] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:28.812492 28128 raft_consensus.cc:482] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:28.812588 28128 raft_consensus.cc:3037] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:11:28.814515 27312 tablet.cc:2369] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:28.814667 28128 raft_consensus.cc:504] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.815310 28128 leader_election.cc:288] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:28.815735 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cb2329d176441ca9127187db2d157a" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:28.815805 27526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cb2329d176441ca9127187db2d157a" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:28.815901 27135 raft_consensus.cc:3037] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:28.816020 27526 raft_consensus.cc:3037] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:28.817888 27135 raft_consensus.cc:2454] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 1.
I20240501 20:11:28.818080 27526 raft_consensus.cc:2454] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 1.
I20240501 20:11:28.818251 27201 leader_election.cc:302] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:28.818567 28128 raft_consensus.cc:2781] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:28.818996 28128 raft_consensus.cc:686] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Becoming Leader. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Running, Role: LEADER
I20240501 20:11:28.819437 28128 consensus_queue.cc:237] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:28.823395 26993 catalog_manager.cc:5484] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8b307156e1346bd9eecf655f49924c7 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
W20240501 20:11:28.862939 27181 tablet.cc:2369] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:28.899776 28133 raft_consensus.cc:482] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:28.899981 28133 raft_consensus.cc:504] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.901665 28133 leader_election.cc:288] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:28.908789 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02a619bc1584d53ac902c8d8baff976" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:28.908803 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02a619bc1584d53ac902c8d8baff976" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:28.909041 27396 raft_consensus.cc:2454] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:28.909032 27135 raft_consensus.cc:2454] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:28.909540 27461 leader_election.cc:302] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:28.909910 28133 raft_consensus.cc:2781] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:28.910023 28133 raft_consensus.cc:482] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:28.910121 28133 raft_consensus.cc:3037] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:28.912107 28133 raft_consensus.cc:504] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.912729 28133 leader_election.cc:288] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:28.913128 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02a619bc1584d53ac902c8d8baff976" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:28.913231 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02a619bc1584d53ac902c8d8baff976" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:28.913321 27135 raft_consensus.cc:3037] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:28.913440 27396 raft_consensus.cc:3037] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:28.915133 27135 raft_consensus.cc:2454] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:28.915472 27396 raft_consensus.cc:2454] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:28.915490 27461 leader_election.cc:302] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:28.915768 28133 raft_consensus.cc:2781] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:28.916152 28133 raft_consensus.cc:686] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:28.916601 28133 consensus_queue.cc:237] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:28.920871 26993 catalog_manager.cc:5484] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:11:29.224503 28132 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:29.233422 28132 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:29.296353 28149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:29.296662 28149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:29.304154 28149 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 20:11:29.354390 28128 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:29.381259 28179 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:29.439476 28133 consensus_queue.cc:1035] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:29.440261 27135 raft_consensus.cc:1261] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:29.459986 28194 consensus_queue.cc:1035] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:29.525606 28196 mvcc.cc:205] Tried to move back new op lower bound from 7022978209539022848 to 7022978207406108672. Current Snapshot: MvccSnapshot[applied={T|T < 7022978209539022848}]
I20240501 20:11:29.529224 28153 mvcc.cc:205] Tried to move back new op lower bound from 7022978209539022848 to 7022978207406108672. Current Snapshot: MvccSnapshot[applied={T|T < 7022978209539022848}]
I20240501 20:11:29.557221 28152 mvcc.cc:205] Tried to move back new op lower bound from 7022978209539022848 to 7022978207406108672. Current Snapshot: MvccSnapshot[applied={T|T < 7022978209539022848}]
INSERT report
rows total: 2000
time total: 260.66 ms
time per row: 0.13033 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:11:30.644222 26992 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:30.653343 27376 tablet_service.cc:1467] Processing CreateTablet for tablet af693dc19f844b12973b000093021619 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.653679 27505 tablet_service.cc:1467] Processing CreateTablet for tablet af693dc19f844b12973b000093021619 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.653869 27376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693dc19f844b12973b000093021619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.654106 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693dc19f844b12973b000093021619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.655088 27375 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb13c1dce8640b28063daa0132d2896 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.655441 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb13c1dce8640b28063daa0132d2896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.659327 27506 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb13c1dce8640b28063daa0132d2896 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.659544 27244 tablet_service.cc:1467] Processing CreateTablet for tablet 553d443dfb344b2690c260d207901ff1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.659750 27506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb13c1dce8640b28063daa0132d2896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.659992 27244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d443dfb344b2690c260d207901ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.660049 27246 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb13c1dce8640b28063daa0132d2896 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.659526 27245 tablet_service.cc:1467] Processing CreateTablet for tablet af693dc19f844b12973b000093021619 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.660246 27374 tablet_service.cc:1467] Processing CreateTablet for tablet 553d443dfb344b2690c260d207901ff1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.660382 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb13c1dce8640b28063daa0132d2896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.660586 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693dc19f844b12973b000093021619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.660629 27374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d443dfb344b2690c260d207901ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.663051 27115 tablet_service.cc:1467] Processing CreateTablet for tablet 553d443dfb344b2690c260d207901ff1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:30.663476 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d443dfb344b2690c260d207901ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:30.672916 28226 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:30.674930 28226 tablet_bootstrap.cc:654] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.679111 28228 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:30.681049 28229 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:30.682638 28228 tablet_bootstrap.cc:654] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.683554 28229 tablet_bootstrap.cc:654] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.683619 28226 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:30.683794 28226 ts_tablet_manager.cc:1367] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 20:11:30.684118 28227 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:30.684835 28226 raft_consensus.cc:348] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.685117 28226 raft_consensus.cc:374] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.685285 28226 raft_consensus.cc:729] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.685508 28227 tablet_bootstrap.cc:654] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.685643 28226 consensus_queue.cc:260] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.686571 28226 ts_tablet_manager.cc:1398] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:11:30.686928 28226 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:30.688727 28226 tablet_bootstrap.cc:654] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.689347 28228 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:30.689446 28228 ts_tablet_manager.cc:1367] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 20:11:30.690186 28228 raft_consensus.cc:348] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.690491 28228 raft_consensus.cc:374] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.690645 28228 raft_consensus.cc:729] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.690966 28228 consensus_queue.cc:260] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.691550 28228 ts_tablet_manager.cc:1398] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:30.691864 28228 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:30.692245 28226 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:30.692390 28226 ts_tablet_manager.cc:1367] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240501 20:11:30.692565 28229 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:30.692692 28229 ts_tablet_manager.cc:1367] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240501 20:11:30.693101 28226 raft_consensus.cc:348] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.693311 28226 raft_consensus.cc:374] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.693426 28226 raft_consensus.cc:729] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.693624 28229 raft_consensus.cc:348] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.693697 28226 consensus_queue.cc:260] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.693924 28229 raft_consensus.cc:374] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.694082 28229 raft_consensus.cc:729] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.694207 28226 ts_tablet_manager.cc:1398] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:30.694491 28226 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:30.694913 28229 consensus_queue.cc:260] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.695077 28228 tablet_bootstrap.cc:654] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.695556 28229 ts_tablet_manager.cc:1398] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:11:30.695969 28226 tablet_bootstrap.cc:654] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.696616 28227 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:30.696830 28227 ts_tablet_manager.cc:1367] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 20:11:30.697865 28227 raft_consensus.cc:348] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.698200 28227 raft_consensus.cc:374] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.698357 28227 raft_consensus.cc:729] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.698701 28227 consensus_queue.cc:260] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.699440 28227 ts_tablet_manager.cc:1398] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:30.699874 28227 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:30.702216 28227 tablet_bootstrap.cc:654] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.703871 28226 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:30.704107 28226 ts_tablet_manager.cc:1367] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:11:30.705101 28226 raft_consensus.cc:348] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.705420 28226 raft_consensus.cc:374] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.705487 28228 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:30.705583 28226 raft_consensus.cc:729] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.705711 28228 ts_tablet_manager.cc:1367] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240501 20:11:30.705946 28226 consensus_queue.cc:260] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.706557 28228 raft_consensus.cc:348] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.706905 28228 raft_consensus.cc:374] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.707051 28228 raft_consensus.cc:729] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.707346 28228 consensus_queue.cc:260] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.708087 28228 ts_tablet_manager.cc:1398] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:30.708412 28228 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:30.709800 28239 raft_consensus.cc:482] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:30.710007 28228 tablet_bootstrap.cc:654] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:30.710569 28239 raft_consensus.cc:504] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.710392 28226 ts_tablet_manager.cc:1398] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240501 20:11:30.711970 28227 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:30.712133 28227 ts_tablet_manager.cc:1367] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.000s
I20240501 20:11:30.712807 28227 raft_consensus.cc:348] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.713034 28227 raft_consensus.cc:374] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.713110 28227 raft_consensus.cc:729] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.713359 28227 consensus_queue.cc:260] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.714033 28227 ts_tablet_manager.cc:1398] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:30.714794 28239 leader_election.cc:288] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:30.716486 28228 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:30.716704 28228 ts_tablet_manager.cc:1367] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240501 20:11:30.717693 28228 raft_consensus.cc:348] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.718008 28228 raft_consensus.cc:374] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:30.718166 28228 raft_consensus.cc:729] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:30.718466 28228 consensus_queue.cc:260] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.719270 28228 ts_tablet_manager.cc:1398] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:30.727350 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:30.727584 27135 raft_consensus.cc:2454] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:30.728116 27331 leader_election.cc:302] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:30.728821 28239 raft_consensus.cc:2781] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:30.728966 28239 raft_consensus.cc:482] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:30.729084 28239 raft_consensus.cc:3037] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.729635 28219 raft_consensus.cc:482] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:30.729748 28219 raft_consensus.cc:504] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.730355 28219 leader_election.cc:288] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:30.731945 28239 raft_consensus.cc:504] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.732618 28239 leader_election.cc:288] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:30.733258 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:30.733268 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af693dc19f844b12973b000093021619" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:30.733453 27395 raft_consensus.cc:2454] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:30.733474 27135 raft_consensus.cc:3037] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.733510 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:30.733726 27266 raft_consensus.cc:2454] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:30.734267 27462 leader_election.cc:302] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:30.734647 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:30.734556 28219 raft_consensus.cc:2781] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:30.734841 27266 raft_consensus.cc:3037] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.734917 28219 raft_consensus.cc:482] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:30.735036 28219 raft_consensus.cc:3037] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.735988 27135 raft_consensus.cc:2454] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:30.736848 27331 leader_election.cc:302] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:30.737704 27266 raft_consensus.cc:2454] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:30.737807 28239 raft_consensus.cc:2781] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:30.737833 28219 raft_consensus.cc:504] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.738025 28239 raft_consensus.cc:686] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Becoming Leader. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:11:30.738037 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af693dc19f844b12973b000093021619" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:30.738277 27265 raft_consensus.cc:2454] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:30.738533 28219 leader_election.cc:288] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:30.738545 28239 consensus_queue.cc:237] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.739295 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af693dc19f844b12973b000093021619" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:30.739293 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af693dc19f844b12973b000093021619" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:30.739466 27395 raft_consensus.cc:3037] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.739473 27266 raft_consensus.cc:3037] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.742202 27395 raft_consensus.cc:2454] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:30.742269 26993 catalog_manager.cc:5484] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to e677127814314770b3370764d0ff9d06 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:11:30.742570 27266 raft_consensus.cc:2454] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:30.742616 27462 leader_election.cc:302] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:30.742929 28219 raft_consensus.cc:2781] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:30.744146 28219 raft_consensus.cc:686] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:30.744419 28219 consensus_queue.cc:237] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:30.747013 26993 catalog_manager.cc:5484] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:11:30.916393 28219 raft_consensus.cc:482] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:30.916591 28219 raft_consensus.cc:504] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.917111 28219 leader_election.cc:288] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:30.917583 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb13c1dce8640b28063daa0132d2896" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:30.917613 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb13c1dce8640b28063daa0132d2896" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:30.917819 27266 raft_consensus.cc:2454] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:30.917810 27395 raft_consensus.cc:2454] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:30.918229 27461 leader_election.cc:302] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:30.918535 28219 raft_consensus.cc:2781] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:30.918663 28219 raft_consensus.cc:482] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:30.918772 28219 raft_consensus.cc:3037] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.920640 28219 raft_consensus.cc:504] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.921146 28219 leader_election.cc:288] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:30.921497 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb13c1dce8640b28063daa0132d2896" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:30.921561 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb13c1dce8640b28063daa0132d2896" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:30.921684 27266 raft_consensus.cc:3037] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.921725 27395 raft_consensus.cc:3037] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:30.923559 27395 raft_consensus.cc:2454] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:30.923831 27266 raft_consensus.cc:2454] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:30.923928 27462 leader_election.cc:302] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:30.924189 28219 raft_consensus.cc:2781] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:30.924341 28219 raft_consensus.cc:686] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:30.924578 28219 consensus_queue.cc:237] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:30.927107 26993 catalog_manager.cc:5484] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:11:31.141260 28239 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:31.153834 28239 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:31.248198 28219 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:31.255704 28219 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:31.326293 28250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:31.326594 28250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:31.333851 28250 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:11:31.438769 28219 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:31.455582 28247 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 247.812 ms
time per row: 0.123906 ms
I20240501 20:11:31.728039 26992 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:31.737274 27115 tablet_service.cc:1467] Processing CreateTablet for tablet 74c414c2eed045bda9d85fe5c2fea654 (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.737432 27245 tablet_service.cc:1467] Processing CreateTablet for tablet 74c414c2eed045bda9d85fe5c2fea654 (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.737783 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c414c2eed045bda9d85fe5c2fea654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.737823 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c414c2eed045bda9d85fe5c2fea654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.738855 27244 tablet_service.cc:1467] Processing CreateTablet for tablet 78fcaa70e916442ea9c67cc762c8db6d (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.739300 27244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78fcaa70e916442ea9c67cc762c8db6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.739420 27114 tablet_service.cc:1467] Processing CreateTablet for tablet 309e90bbd9f140a8beca9b3da3f75db4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.739821 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309e90bbd9f140a8beca9b3da3f75db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.743047 27113 tablet_service.cc:1467] Processing CreateTablet for tablet 78fcaa70e916442ea9c67cc762c8db6d (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.743451 27113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78fcaa70e916442ea9c67cc762c8db6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.743608 27506 tablet_service.cc:1467] Processing CreateTablet for tablet 309e90bbd9f140a8beca9b3da3f75db4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.743968 27505 tablet_service.cc:1467] Processing CreateTablet for tablet 74c414c2eed045bda9d85fe5c2fea654 (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.744694 27375 tablet_service.cc:1467] Processing CreateTablet for tablet 309e90bbd9f140a8beca9b3da3f75db4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.745079 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309e90bbd9f140a8beca9b3da3f75db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.744061 27506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309e90bbd9f140a8beca9b3da3f75db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.747138 27504 tablet_service.cc:1467] Processing CreateTablet for tablet 78fcaa70e916442ea9c67cc762c8db6d (DEFAULT_TABLE table=rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:31.747581 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78fcaa70e916442ea9c67cc762c8db6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.752521 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c414c2eed045bda9d85fe5c2fea654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:31.756662 28312 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:31.758597 28312 tablet_bootstrap.cc:654] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.764207 28315 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:31.769023 28315 tablet_bootstrap.cc:654] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.770319 28314 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:31.771869 28311 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:31.772835 28314 tablet_bootstrap.cc:654] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.773273 28312 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:31.773485 28312 ts_tablet_manager.cc:1367] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20240501 20:11:31.773540 28315 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:31.773787 28315 ts_tablet_manager.cc:1367] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 20:11:31.774242 28311 tablet_bootstrap.cc:654] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.774608 28312 raft_consensus.cc:348] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:31.774789 28315 raft_consensus.cc:348] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:31.774999 28312 raft_consensus.cc:374] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.775115 28312 raft_consensus.cc:729] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.775147 28315 raft_consensus.cc:374] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.775300 28315 raft_consensus.cc:729] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.775569 28312 consensus_queue.cc:260] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:31.775699 28315 consensus_queue.cc:260] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:31.776577 28315 ts_tablet_manager.cc:1398] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:31.777740 28315 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:31.781123 28315 tablet_bootstrap.cc:654] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.781155 28314 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:31.781329 28314 ts_tablet_manager.cc:1367] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 20:11:31.782282 28314 raft_consensus.cc:348] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.782575 28314 raft_consensus.cc:374] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.782748 28314 raft_consensus.cc:729] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.783073 28314 consensus_queue.cc:260] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.783703 28314 ts_tablet_manager.cc:1398] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:31.785307 28312 ts_tablet_manager.cc:1398] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240501 20:11:31.785665 28312 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:31.787470 28312 tablet_bootstrap.cc:654] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.789361 28312 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:31.789590 28312 ts_tablet_manager.cc:1367] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:31.789636 28311 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:31.789852 28311 ts_tablet_manager.cc:1367] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.013s
I20240501 20:11:31.790082 28315 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:31.790273 28315 ts_tablet_manager.cc:1367] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240501 20:11:31.790515 28312 raft_consensus.cc:348] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:31.790771 28312 raft_consensus.cc:374] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.790889 28312 raft_consensus.cc:729] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.790794 28311 raft_consensus.cc:348] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.791006 28311 raft_consensus.cc:374] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.791107 28311 raft_consensus.cc:729] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.791191 28312 consensus_queue.cc:260] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:31.791216 28315 raft_consensus.cc:348] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:31.791428 28315 raft_consensus.cc:374] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.791339 28311 consensus_queue.cc:260] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.791512 28315 raft_consensus.cc:729] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.791859 28311 ts_tablet_manager.cc:1398] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:31.791747 28315 consensus_queue.cc:260] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:31.792152 28311 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:31.792273 28315 ts_tablet_manager.cc:1398] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:31.792589 28315 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:31.793856 28311 tablet_bootstrap.cc:654] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.794215 28312 ts_tablet_manager.cc:1398] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:11:31.794241 28315 tablet_bootstrap.cc:654] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.796275 28315 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:31.796463 28315 ts_tablet_manager.cc:1367] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:31.797605 28311 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:31.797672 28315 raft_consensus.cc:348] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.797820 28311 ts_tablet_manager.cc:1367] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:11:31.797890 28315 raft_consensus.cc:374] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.798002 28315 raft_consensus.cc:729] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.798228 28315 consensus_queue.cc:260] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.798591 28311 raft_consensus.cc:348] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:31.798844 28311 raft_consensus.cc:374] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.798928 28311 raft_consensus.cc:729] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.799175 28311 consensus_queue.cc:260] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:31.799338 28315 ts_tablet_manager.cc:1398] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:31.799721 28311 ts_tablet_manager.cc:1398] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:31.800007 28311 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:31.801501 28311 tablet_bootstrap.cc:654] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:31.803723 28311 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:31.803862 28311 ts_tablet_manager.cc:1367] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:31.804445 28311 raft_consensus.cc:348] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:31.804598 28311 raft_consensus.cc:374] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:31.804698 28311 raft_consensus.cc:729] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:31.804886 28311 consensus_queue.cc:260] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:31.805341 28311 ts_tablet_manager.cc:1398] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:31.858759 28220 raft_consensus.cc:482] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:31.858927 28220 raft_consensus.cc:504] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.859791 28220 leader_election.cc:288] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:31.860414 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309e90bbd9f140a8beca9b3da3f75db4" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:31.860702 27394 raft_consensus.cc:2454] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:31.861161 27071 leader_election.cc:302] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:31.861478 28220 raft_consensus.cc:2781] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:31.861653 28220 raft_consensus.cc:482] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:31.861769 28220 raft_consensus.cc:3037] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:31.864425 28220 raft_consensus.cc:504] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.865046 28220 leader_election.cc:288] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:31.865638 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309e90bbd9f140a8beca9b3da3f75db4" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:31.865857 27394 raft_consensus.cc:3037] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:31.867190 27526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309e90bbd9f140a8beca9b3da3f75db4" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:31.867467 27526 raft_consensus.cc:2454] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:31.867386 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309e90bbd9f140a8beca9b3da3f75db4" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:31.867733 27525 raft_consensus.cc:3037] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:31.868687 27394 raft_consensus.cc:2454] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:31.869240 27071 leader_election.cc:302] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:31.869545 28220 raft_consensus.cc:2781] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:31.870899 27525 raft_consensus.cc:2454] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:31.871479 28220 raft_consensus.cc:686] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:31.872084 28220 consensus_queue.cc:237] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:31.875317 26988 catalog_manager.cc:5484] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:11:32.019376 28220 raft_consensus.cc:482] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:32.019634 28220 raft_consensus.cc:504] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:32.020473 28220 leader_election.cc:288] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:32.021064 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78fcaa70e916442ea9c67cc762c8db6d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:32.021080 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78fcaa70e916442ea9c67cc762c8db6d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:32.021297 27266 raft_consensus.cc:2454] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:32.021312 27525 raft_consensus.cc:2454] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:32.021771 27069 leader_election.cc:302] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:32.022125 28220 raft_consensus.cc:2781] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:32.022221 28220 raft_consensus.cc:482] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:32.022312 28220 raft_consensus.cc:3037] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:32.025108 28220 raft_consensus.cc:504] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:32.025826 28220 leader_election.cc:288] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:32.026350 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78fcaa70e916442ea9c67cc762c8db6d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:32.026343 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78fcaa70e916442ea9c67cc762c8db6d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:32.026558 27525 raft_consensus.cc:3037] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:32.026553 27266 raft_consensus.cc:3037] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:32.028512 27266 raft_consensus.cc:2454] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:32.028944 27070 leader_election.cc:302] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:32.029228 28220 raft_consensus.cc:2781] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:32.029258 27525 raft_consensus.cc:2454] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:32.029500 28220 raft_consensus.cc:686] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:32.029774 28220 consensus_queue.cc:237] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:32.032754 26988 catalog_manager.cc:5484] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:11:32.109910 28259 raft_consensus.cc:482] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:32.110095 28259 raft_consensus.cc:504] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:32.110759 28259 leader_election.cc:288] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:32.111328 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c414c2eed045bda9d85fe5c2fea654" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:32.111356 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c414c2eed045bda9d85fe5c2fea654" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:32.111596 27134 raft_consensus.cc:2454] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:32.111611 27266 raft_consensus.cc:2454] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:32.111999 27461 leader_election.cc:302] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:32.112282 28259 raft_consensus.cc:2781] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:32.112396 28259 raft_consensus.cc:482] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:32.112488 28259 raft_consensus.cc:3037] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:32.115279 28259 raft_consensus.cc:504] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:32.116219 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c414c2eed045bda9d85fe5c2fea654" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:32.116307 28259 leader_election.cc:288] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:32.116428 27134 raft_consensus.cc:3037] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:32.116593 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c414c2eed045bda9d85fe5c2fea654" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:32.116802 27266 raft_consensus.cc:3037] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:32.118284 27134 raft_consensus.cc:2454] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:32.118786 27461 leader_election.cc:302] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:32.119086 28259 raft_consensus.cc:2781] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:32.119248 28259 raft_consensus.cc:686] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:32.119535 27266 raft_consensus.cc:2454] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:32.119504 28259 consensus_queue.cc:237] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:32.122844 26988 catalog_manager.cc:5484] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:11:32.434290 28220 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:32.440754 28220 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:32.472576 28220 consensus_queue.cc:1035] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:32.477094 28220 consensus_queue.cc:1035] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:32.497131 28259 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:32.519868 28259 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:32.569778 28330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:32.570078 28330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:32.577311 28330 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 242.869 ms
time per row: 0.121435 ms
I20240501 20:11:32.965160 26985 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:32.974184 27375 tablet_service.cc:1467] Processing CreateTablet for tablet b5427fbc8a244a69974fd8d0f5f3d08d (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.974412 27114 tablet_service.cc:1467] Processing CreateTablet for tablet b5427fbc8a244a69974fd8d0f5f3d08d (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.974699 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5427fbc8a244a69974fd8d0f5f3d08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.974807 27376 tablet_service.cc:1467] Processing CreateTablet for tablet f034a679870b44caa238f04a8b49d302 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.974901 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5427fbc8a244a69974fd8d0f5f3d08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.975122 27376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f034a679870b44caa238f04a8b49d302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.976742 27244 tablet_service.cc:1467] Processing CreateTablet for tablet 17e69dbd245e4b8db397aaffc9145c9f (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.977147 27244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e69dbd245e4b8db397aaffc9145c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.978052 27506 tablet_service.cc:1467] Processing CreateTablet for tablet f034a679870b44caa238f04a8b49d302 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.978492 27506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f034a679870b44caa238f04a8b49d302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.981606 27115 tablet_service.cc:1467] Processing CreateTablet for tablet f034a679870b44caa238f04a8b49d302 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.982812 27374 tablet_service.cc:1467] Processing CreateTablet for tablet 17e69dbd245e4b8db397aaffc9145c9f (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.982976 27505 tablet_service.cc:1467] Processing CreateTablet for tablet 17e69dbd245e4b8db397aaffc9145c9f (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.983167 27374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e69dbd245e4b8db397aaffc9145c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.983312 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e69dbd245e4b8db397aaffc9145c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.987049 27245 tablet_service.cc:1467] Processing CreateTablet for tablet b5427fbc8a244a69974fd8d0f5f3d08d (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:32.987468 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5427fbc8a244a69974fd8d0f5f3d08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.982010 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f034a679870b44caa238f04a8b49d302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.989725 28394 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:32.992142 28396 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:32.992295 28394 tablet_bootstrap.cc:654] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:32.994993 28396 tablet_bootstrap.cc:654] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.000043 28395 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:33.002491 28395 tablet_bootstrap.cc:654] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.007567 28396 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:33.007720 28396 ts_tablet_manager.cc:1367] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20240501 20:11:33.008771 28396 raft_consensus.cc:348] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.009033 28396 raft_consensus.cc:374] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.009127 28396 raft_consensus.cc:729] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.009342 28400 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:33.009394 28396 consensus_queue.cc:260] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.010058 28396 ts_tablet_manager.cc:1398] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:33.010392 28396 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:33.010876 28394 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:33.011030 28394 ts_tablet_manager.cc:1367] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.010s
I20240501 20:11:33.012102 28396 tablet_bootstrap.cc:654] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.012023 28394 raft_consensus.cc:348] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.012259 28394 raft_consensus.cc:374] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.012382 28394 raft_consensus.cc:729] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.012701 28394 consensus_queue.cc:260] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.013415 28394 ts_tablet_manager.cc:1398] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:33.013608 28400 tablet_bootstrap.cc:654] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.013779 28394 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:33.015870 28394 tablet_bootstrap.cc:654] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.022518 28400 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:33.022720 28400 ts_tablet_manager.cc:1367] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240501 20:11:33.024756 28400 raft_consensus.cc:348] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.025081 28400 raft_consensus.cc:374] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.025224 28400 raft_consensus.cc:729] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.025514 28400 consensus_queue.cc:260] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.027164 28400 ts_tablet_manager.cc:1398] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:11:33.027460 28400 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:33.029014 28400 tablet_bootstrap.cc:654] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.035806 28395 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:33.036080 28395 ts_tablet_manager.cc:1367] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.005s
I20240501 20:11:33.037150 28395 raft_consensus.cc:348] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.037458 28395 raft_consensus.cc:374] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.037619 28395 raft_consensus.cc:729] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.037987 28400 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:33.037930 28395 consensus_queue.cc:260] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.038190 28400 ts_tablet_manager.cc:1367] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240501 20:11:33.038990 28400 raft_consensus.cc:348] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.039279 28400 raft_consensus.cc:374] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.039400 28400 raft_consensus.cc:729] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.039639 28395 ts_tablet_manager.cc:1398] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:11:33.039683 28400 consensus_queue.cc:260] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.040012 28395 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:33.040267 28400 ts_tablet_manager.cc:1398] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:33.042881 28395 tablet_bootstrap.cc:654] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.051105 28394 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:33.051105 28396 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:33.051234 28395 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:33.051301 28394 ts_tablet_manager.cc:1367] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.027s
I20240501 20:11:33.051296 28396 ts_tablet_manager.cc:1367] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.018s
I20240501 20:11:33.051474 28395 ts_tablet_manager.cc:1367] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 20:11:33.052111 28394 raft_consensus.cc:348] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.052111 28396 raft_consensus.cc:348] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.052354 28396 raft_consensus.cc:374] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.052218 28395 raft_consensus.cc:348] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.052408 28394 raft_consensus.cc:374] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.052481 28396 raft_consensus.cc:729] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.052532 28395 raft_consensus.cc:374] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.052524 28394 raft_consensus.cc:729] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.052634 28395 raft_consensus.cc:729] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.052817 28394 consensus_queue.cc:260] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.052898 28395 consensus_queue.cc:260] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.052878 28396 consensus_queue.cc:260] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.053411 28394 ts_tablet_manager.cc:1398] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:33.053612 28396 ts_tablet_manager.cc:1398] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:33.054203 28395 ts_tablet_manager.cc:1398] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:11:33.054556 28395 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:33.056824 28395 tablet_bootstrap.cc:654] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.059738 28395 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:33.059875 28395 ts_tablet_manager.cc:1367] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 20:11:33.060442 28395 raft_consensus.cc:348] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.060598 28395 raft_consensus.cc:374] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.060696 28395 raft_consensus.cc:729] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.060885 28395 consensus_queue.cc:260] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.061355 28395 ts_tablet_manager.cc:1398] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:33.157032 28328 raft_consensus.cc:482] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:33.157183 28328 raft_consensus.cc:504] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.157773 28328 leader_election.cc:288] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:33.158269 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f034a679870b44caa238f04a8b49d302" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:33.158308 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f034a679870b44caa238f04a8b49d302" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:33.158555 27134 raft_consensus.cc:2454] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:33.158555 27394 raft_consensus.cc:2454] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:33.159040 27462 leader_election.cc:302] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:33.159345 28328 raft_consensus.cc:2781] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:33.159463 28328 raft_consensus.cc:482] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:33.159565 28328 raft_consensus.cc:3037] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.161500 28328 raft_consensus.cc:504] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.162027 28328 leader_election.cc:288] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:33.162616 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f034a679870b44caa238f04a8b49d302" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:33.162593 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f034a679870b44caa238f04a8b49d302" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:33.162848 27394 raft_consensus.cc:3037] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.162848 27134 raft_consensus.cc:3037] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.165620 27394 raft_consensus.cc:2454] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:33.165619 27134 raft_consensus.cc:2454] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:33.166049 27462 leader_election.cc:302] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:33.166365 28328 raft_consensus.cc:2781] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:33.166536 28328 raft_consensus.cc:686] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:33.166846 28328 consensus_queue.cc:237] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:33.169710 26985 catalog_manager.cc:5484] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:11:33.251766 28325 raft_consensus.cc:482] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:33.251977 28325 raft_consensus.cc:504] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.252614 28325 leader_election.cc:288] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:33.253116 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5427fbc8a244a69974fd8d0f5f3d08d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:33.253129 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5427fbc8a244a69974fd8d0f5f3d08d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:33.253330 27266 raft_consensus.cc:2454] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:33.253324 27394 raft_consensus.cc:2454] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:33.253760 27071 leader_election.cc:302] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:33.254074 28325 raft_consensus.cc:2781] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:33.254191 28325 raft_consensus.cc:482] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:33.254257 28325 raft_consensus.cc:3037] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.256140 28325 raft_consensus.cc:504] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.256642 28325 leader_election.cc:288] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:33.257165 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5427fbc8a244a69974fd8d0f5f3d08d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:33.257210 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5427fbc8a244a69974fd8d0f5f3d08d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:33.257345 27394 raft_consensus.cc:3037] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.257378 27266 raft_consensus.cc:3037] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.259302 27266 raft_consensus.cc:2454] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:33.259387 27394 raft_consensus.cc:2454] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:33.259683 27070 leader_election.cc:302] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:33.259992 28325 raft_consensus.cc:2781] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:33.260138 28325 raft_consensus.cc:686] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:33.260411 28325 consensus_queue.cc:237] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:33.262468 28328 raft_consensus.cc:482] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:33.262679 28328 raft_consensus.cc:504] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.263406 28328 leader_election.cc:288] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:33.263413 26985 catalog_manager.cc:5484] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:11:33.263971 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:33.263998 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:33.264191 27394 raft_consensus.cc:2454] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:33.264191 27265 raft_consensus.cc:2454] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:33.264592 27462 leader_election.cc:302] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:33.265197 28328 raft_consensus.cc:2781] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:33.265319 28328 raft_consensus.cc:482] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:33.265450 28328 raft_consensus.cc:3037] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.267504 28328 raft_consensus.cc:504] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.268105 28328 leader_election.cc:288] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:33.268445 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:33.268543 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:33.268633 27265 raft_consensus.cc:3037] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.268759 27394 raft_consensus.cc:3037] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.270524 27265 raft_consensus.cc:2454] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:33.270648 27394 raft_consensus.cc:2454] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:33.270911 27461 leader_election.cc:302] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:33.271165 28328 raft_consensus.cc:2781] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:33.271339 28328 raft_consensus.cc:686] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:33.271595 28328 consensus_queue.cc:237] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:33.274120 26985 catalog_manager.cc:5484] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
W20240501 20:11:33.632738 28410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:33.633051 28410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:33.640694 28410 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:33.659914 28328 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:33.666551 28328 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 20:11:33.689942 28328 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:33.694801 28259 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:33.782555 27394 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:33.782578 27265 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:33.783257 28325 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:33.783943 28325 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 205.836 ms
time per row: 0.102918 ms
I20240501 20:11:33.998901 26993 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:34.008015 27115 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8902b7b0b649edb10f362713ad36fe (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.008116 27245 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8902b7b0b649edb10f362713ad36fe (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.008522 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8902b7b0b649edb10f362713ad36fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.008535 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8902b7b0b649edb10f362713ad36fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.008527 27244 tablet_service.cc:1467] Processing CreateTablet for tablet c0e88937e1ec49bfbaffe535e445e40a (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.008919 27244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e88937e1ec49bfbaffe535e445e40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.013911 28476 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:34.014072 27505 tablet_service.cc:1467] Processing CreateTablet for tablet c0e88937e1ec49bfbaffe535e445e40a (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.014493 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e88937e1ec49bfbaffe535e445e40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.015995 28476 tablet_bootstrap.cc:654] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.019582 28477 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:34.019738 27375 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8902b7b0b649edb10f362713ad36fe (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.020328 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8902b7b0b649edb10f362713ad36fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.022468 28477 tablet_bootstrap.cc:654] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.024823 27376 tablet_service.cc:1467] Processing CreateTablet for tablet b5a20d47052f404986315ddb05284ad9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.025226 27376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a20d47052f404986315ddb05284ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.026840 27506 tablet_service.cc:1467] Processing CreateTablet for tablet b5a20d47052f404986315ddb05284ad9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.027210 27506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a20d47052f404986315ddb05284ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.031332 27114 tablet_service.cc:1467] Processing CreateTablet for tablet b5a20d47052f404986315ddb05284ad9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.031745 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a20d47052f404986315ddb05284ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.032243 27115 tablet_service.cc:1467] Processing CreateTablet for tablet c0e88937e1ec49bfbaffe535e445e40a (DEFAULT_TABLE table=rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:34.032600 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e88937e1ec49bfbaffe535e445e40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:34.042061 28481 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:34.043048 28480 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:34.044648 28481 tablet_bootstrap.cc:654] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.044697 28480 tablet_bootstrap.cc:654] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.048219 28481 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:34.048233 28477 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:34.048365 28477 ts_tablet_manager.cc:1367] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20240501 20:11:34.048369 28481 ts_tablet_manager.cc:1367] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:11:34.049126 28476 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:34.049300 28476 ts_tablet_manager.cc:1367] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20240501 20:11:34.049293 28481 raft_consensus.cc:348] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.049592 28481 raft_consensus.cc:374] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.049724 28481 raft_consensus.cc:729] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.050004 28481 consensus_queue.cc:260] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.050148 28476 raft_consensus.cc:348] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.050386 28476 raft_consensus.cc:374] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.050523 28476 raft_consensus.cc:729] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.050928 28481 ts_tablet_manager.cc:1398] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:34.050868 28476 consensus_queue.cc:260] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.051064 28477 raft_consensus.cc:348] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.051319 28481 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:34.051352 28477 raft_consensus.cc:374] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.051512 28477 raft_consensus.cc:729] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.052198 28476 ts_tablet_manager.cc:1398] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:34.052462 28476 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:34.052996 28481 tablet_bootstrap.cc:654] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.053512 28477 consensus_queue.cc:260] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.055181 28476 tablet_bootstrap.cc:654] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.056826 28477 ts_tablet_manager.cc:1398] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:11:34.057188 28477 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:34.062551 28477 tablet_bootstrap.cc:654] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.064570 28480 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:34.064795 28480 ts_tablet_manager.cc:1367] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 20:11:34.065598 28480 raft_consensus.cc:348] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.065886 28480 raft_consensus.cc:374] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.066048 28480 raft_consensus.cc:729] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.066411 28480 consensus_queue.cc:260] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.067204 28480 ts_tablet_manager.cc:1398] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:34.067622 28477 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:34.067795 28477 ts_tablet_manager.cc:1367] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:11:34.068701 28477 raft_consensus.cc:348] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.068993 28477 raft_consensus.cc:374] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.069077 28477 raft_consensus.cc:729] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.069362 28477 consensus_queue.cc:260] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.070238 28477 ts_tablet_manager.cc:1398] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:34.070591 28477 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:34.066823 28476 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:34.070852 28476 ts_tablet_manager.cc:1367] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.004s
I20240501 20:11:34.071687 28476 raft_consensus.cc:348] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.071931 28476 raft_consensus.cc:374] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.072005 28476 raft_consensus.cc:729] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.072264 28476 consensus_queue.cc:260] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.072851 28476 ts_tablet_manager.cc:1398] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:34.073247 28477 tablet_bootstrap.cc:654] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.075567 28481 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:34.075786 28481 ts_tablet_manager.cc:1367] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.008s
I20240501 20:11:34.076215 28477 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:34.076380 28477 ts_tablet_manager.cc:1367] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:11:34.076632 28481 raft_consensus.cc:348] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.076905 28481 raft_consensus.cc:374] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.077023 28481 raft_consensus.cc:729] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.077198 28477 raft_consensus.cc:348] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.077435 28477 raft_consensus.cc:374] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.077380 28481 consensus_queue.cc:260] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.077553 28477 raft_consensus.cc:729] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.077806 28477 consensus_queue.cc:260] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.078482 28477 ts_tablet_manager.cc:1398] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:34.078747 28481 ts_tablet_manager.cc:1398] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:34.080724 28480 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:34.076594 28453 raft_consensus.cc:482] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:34.082322 28453 raft_consensus.cc:504] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.082964 28480 tablet_bootstrap.cc:654] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:34.083106 28453 leader_election.cc:288] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:34.084110 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e88937e1ec49bfbaffe535e445e40a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:34.084300 27134 raft_consensus.cc:2454] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:34.084820 27461 leader_election.cc:302] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:34.086223 28464 raft_consensus.cc:2781] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:34.086321 28464 raft_consensus.cc:482] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:34.086416 28464 raft_consensus.cc:3037] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.088380 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e88937e1ec49bfbaffe535e445e40a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:34.088624 27266 raft_consensus.cc:2454] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:34.089114 28480 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:34.089322 28480 ts_tablet_manager.cc:1367] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20240501 20:11:34.089290 28464 raft_consensus.cc:504] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.090130 28480 raft_consensus.cc:348] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.090446 28480 raft_consensus.cc:374] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:34.090587 28480 raft_consensus.cc:729] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:34.090879 28464 leader_election.cc:288] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:34.090939 28480 consensus_queue.cc:260] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.091115 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e88937e1ec49bfbaffe535e445e40a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:34.091326 27266 raft_consensus.cc:3037] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.091274 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e88937e1ec49bfbaffe535e445e40a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:34.091461 27134 raft_consensus.cc:3037] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.091632 28480 ts_tablet_manager.cc:1398] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:34.093431 27266 raft_consensus.cc:2454] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:34.093850 27461 leader_election.cc:302] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:34.093973 27134 raft_consensus.cc:2454] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:34.094152 28464 raft_consensus.cc:2781] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:34.094326 28464 raft_consensus.cc:686] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:34.094563 28464 consensus_queue.cc:237] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.096884 26993 catalog_manager.cc:5484] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
I20240501 20:11:34.109592 28325 raft_consensus.cc:482] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:34.109784 28325 raft_consensus.cc:504] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.110416 28325 leader_election.cc:288] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:34.110975 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8902b7b0b649edb10f362713ad36fe" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:34.110963 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8902b7b0b649edb10f362713ad36fe" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:34.111215 27394 raft_consensus.cc:2454] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:34.111433 27266 raft_consensus.cc:2454] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:34.111665 27071 leader_election.cc:302] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:34.112043 28325 raft_consensus.cc:2781] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:34.112154 28325 raft_consensus.cc:482] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:34.112311 28325 raft_consensus.cc:3037] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.114229 28325 raft_consensus.cc:504] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.114867 28325 leader_election.cc:288] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:34.115218 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8902b7b0b649edb10f362713ad36fe" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:34.115411 27266 raft_consensus.cc:3037] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.115358 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8902b7b0b649edb10f362713ad36fe" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:34.115582 27394 raft_consensus.cc:3037] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.117313 27266 raft_consensus.cc:2454] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:34.117709 27070 leader_election.cc:302] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:34.117946 28325 raft_consensus.cc:2781] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:34.118176 28325 raft_consensus.cc:686] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:34.118399 27394 raft_consensus.cc:2454] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:34.118441 28325 consensus_queue.cc:237] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:34.121299 26993 catalog_manager.cc:5484] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:11:34.123554 28254 raft_consensus.cc:482] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:34.123708 28254 raft_consensus.cc:504] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.124540 28254 leader_election.cc:288] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:34.125722 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a20d47052f404986315ddb05284ad9" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:34.125977 27134 raft_consensus.cc:2454] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:34.126456 27331 leader_election.cc:302] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:34.126756 28254 raft_consensus.cc:2781] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:34.126919 28254 raft_consensus.cc:482] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:34.127040 28254 raft_consensus.cc:3037] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.129110 28254 raft_consensus.cc:504] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.129585 28254 leader_election.cc:288] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:34.134512 27526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a20d47052f404986315ddb05284ad9" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:34.134865 27526 raft_consensus.cc:2454] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:34.135408 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a20d47052f404986315ddb05284ad9" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:34.135637 27525 raft_consensus.cc:3037] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.139230 27525 raft_consensus.cc:2454] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:34.139458 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a20d47052f404986315ddb05284ad9" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:34.139693 27135 raft_consensus.cc:3037] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:34.139842 27330 leader_election.cc:302] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:34.140148 28254 raft_consensus.cc:2781] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:34.140296 28254 raft_consensus.cc:686] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Becoming Leader. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:11:34.140615 28254 consensus_queue.cc:237] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:34.142284 27135 raft_consensus.cc:2454] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:34.143482 26993 catalog_manager.cc:5484] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to e677127814314770b3370764d0ff9d06 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
I20240501 20:11:34.472046 28453 consensus_queue.cc:1035] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:34.485147 28453 consensus_queue.cc:1035] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:34.500802 28494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:34.501154 28494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:11:34.504637 28325 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:34.514719 28494 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:34.522912 28455 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240501 20:11:34.578469 28254 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:34.594179 28492 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 289.207 ms
time per row: 0.144604 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:11:35.919889 26990 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:35.929155 27505 tablet_service.cc:1467] Processing CreateTablet for tablet c619a2235c9248eeb6b2e1d78583d1ba (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.929656 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c619a2235c9248eeb6b2e1d78583d1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.929728 27244 tablet_service.cc:1467] Processing CreateTablet for tablet c619a2235c9248eeb6b2e1d78583d1ba (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.930141 27244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c619a2235c9248eeb6b2e1d78583d1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.931105 27114 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf74c24f28546bb90e10002270e3814 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.931476 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf74c24f28546bb90e10002270e3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.933986 27246 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf74c24f28546bb90e10002270e3814 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.934367 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf74c24f28546bb90e10002270e3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.936810 28560 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:35.938678 28560 tablet_bootstrap.cc:654] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:35.938659 28561 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:35.938584 27506 tablet_service.cc:1467] Processing CreateTablet for tablet 6c836d9afa0f439b978bc2fe5b5e499d (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.939102 27506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c836d9afa0f439b978bc2fe5b5e499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.940115 27245 tablet_service.cc:1467] Processing CreateTablet for tablet 6c836d9afa0f439b978bc2fe5b5e499d (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.940529 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c836d9afa0f439b978bc2fe5b5e499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.941651 28561 tablet_bootstrap.cc:654] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:35.943526 27375 tablet_service.cc:1467] Processing CreateTablet for tablet c619a2235c9248eeb6b2e1d78583d1ba (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.943617 27376 tablet_service.cc:1467] Processing CreateTablet for tablet 6c836d9afa0f439b978bc2fe5b5e499d (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.944182 27374 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf74c24f28546bb90e10002270e3814 (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.945104 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c619a2235c9248eeb6b2e1d78583d1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.949468 27374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf74c24f28546bb90e10002270e3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.950784 27376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c836d9afa0f439b978bc2fe5b5e499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.960487 28564 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:35.963037 28565 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:35.964848 28564 tablet_bootstrap.cc:654] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:35.965116 28565 tablet_bootstrap.cc:654] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:35.974102 28564 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:35.974332 28564 ts_tablet_manager.cc:1367] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240501 20:11:35.975229 28564 raft_consensus.cc:348] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:35.975492 28560 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:35.975551 28564 raft_consensus.cc:374] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:35.975665 28564 raft_consensus.cc:729] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:35.975853 28565 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:35.975831 28560 ts_tablet_manager.cc:1367] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.017s
I20240501 20:11:35.975977 28561 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:35.976042 28565 ts_tablet_manager.cc:1367] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240501 20:11:35.975939 28564 consensus_queue.cc:260] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:35.976171 28561 ts_tablet_manager.cc:1367] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.022s
I20240501 20:11:35.976809 28565 raft_consensus.cc:348] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:35.977054 28565 raft_consensus.cc:374] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:35.976977 28561 raft_consensus.cc:348] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:35.977181 28565 raft_consensus.cc:729] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:35.977228 28564 ts_tablet_manager.cc:1398] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:11:35.977264 28561 raft_consensus.cc:374] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:35.977381 28561 raft_consensus.cc:729] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:35.977432 28565 consensus_queue.cc:260] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:35.976836 28560 raft_consensus.cc:348] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:35.977728 28560 raft_consensus.cc:374] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:35.977694 28561 consensus_queue.cc:260] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:35.977831 28560 raft_consensus.cc:729] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:35.978343 28565 ts_tablet_manager.cc:1398] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:35.978432 28561 ts_tablet_manager.cc:1398] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:35.978765 28561 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:35.979202 28565 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:35.980103 28560 consensus_queue.cc:260] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:35.981647 28565 tablet_bootstrap.cc:654] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:35.981679 28560 ts_tablet_manager.cc:1398] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 20:11:35.982029 28560 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:35.982100 28546 raft_consensus.cc:482] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:35.982252 28546 raft_consensus.cc:504] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:35.983039 28546 leader_election.cc:288] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:35.983651 28560 tablet_bootstrap.cc:654] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:35.983795 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:35.984055 27395 raft_consensus.cc:2454] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:35.984993 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:35.986044 27071 leader_election.cc:302] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
W20240501 20:11:35.986608 27070 leader_election.cc:341] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:35.987684 28546 raft_consensus.cc:2781] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:35.987816 28546 raft_consensus.cc:482] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:35.987922 28546 raft_consensus.cc:3037] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:35.988270 28561 tablet_bootstrap.cc:654] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:35.995146 28546 raft_consensus.cc:504] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:35.996528 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:35.996796 27395 raft_consensus.cc:3037] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:35.997267 28561 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:35.997516 28561 ts_tablet_manager.cc:1367] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20240501 20:11:35.998553 28561 raft_consensus.cc:348] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:35.998889 28561 raft_consensus.cc:374] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:35.998988 28561 raft_consensus.cc:729] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:35.999343 28561 consensus_queue.cc:260] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.000113 28561 ts_tablet_manager.cc:1398] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:11:36.000561 27395 raft_consensus.cc:2454] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:36.000975 28546 leader_election.cc:288] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:36.001341 27071 leader_election.cc:302] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:36.001405 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:36.001742 28546 raft_consensus.cc:2781] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
W20240501 20:11:36.002146 27070 leader_election.cc:341] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:36.008675 28546 raft_consensus.cc:686] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:36.009641 28546 consensus_queue.cc:237] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:36.012001 28560 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:36.011992 28565 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:36.012179 28565 ts_tablet_manager.cc:1367] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.028s
I20240501 20:11:36.012389 28560 ts_tablet_manager.cc:1367] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.015s
I20240501 20:11:36.012990 28565 raft_consensus.cc:348] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.013223 28565 raft_consensus.cc:374] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.013305 28565 raft_consensus.cc:729] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.013208 28560 raft_consensus.cc:348] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:36.013446 28560 raft_consensus.cc:374] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.013530 28560 raft_consensus.cc:729] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.013301 26990 catalog_manager.cc:5484] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:11:36.013607 28565 consensus_queue.cc:260] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.013784 28560 consensus_queue.cc:260] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:36.014153 28565 ts_tablet_manager.cc:1398] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:36.014315 28560 ts_tablet_manager.cc:1398] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:36.014484 28565 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:36.014668 28560 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:36.016237 28560 tablet_bootstrap.cc:654] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.016919 28565 tablet_bootstrap.cc:654] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.036639 28560 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:36.036846 28560 ts_tablet_manager.cc:1367] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.014s
I20240501 20:11:36.036962 28565 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:36.037128 28565 ts_tablet_manager.cc:1367] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.017s
I20240501 20:11:36.038039 28565 raft_consensus.cc:348] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.038267 28565 raft_consensus.cc:374] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.038384 28565 raft_consensus.cc:729] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.038421 28560 raft_consensus.cc:348] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.038676 28560 raft_consensus.cc:374] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.038785 28560 raft_consensus.cc:729] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.038692 28565 consensus_queue.cc:260] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.039028 28560 consensus_queue.cc:260] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.039448 28565 ts_tablet_manager.cc:1398] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:11:36.039572 28560 ts_tablet_manager.cc:1398] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:36.051120 28464 raft_consensus.cc:482] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:36.051283 28464 raft_consensus.cc:504] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.051985 28464 leader_election.cc:288] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:36.052408 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c836d9afa0f439b978bc2fe5b5e499d" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:36.052486 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c836d9afa0f439b978bc2fe5b5e499d" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:36.052655 27264 raft_consensus.cc:2454] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:36.052693 27395 raft_consensus.cc:2454] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:36.053081 27462 leader_election.cc:302] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:36.053324 28464 raft_consensus.cc:2781] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:36.053448 28464 raft_consensus.cc:482] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:36.053539 28464 raft_consensus.cc:3037] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:36.055847 28464 raft_consensus.cc:504] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.056439 28464 leader_election.cc:288] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:36.056845 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c836d9afa0f439b978bc2fe5b5e499d" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:36.056912 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c836d9afa0f439b978bc2fe5b5e499d" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:36.057071 27264 raft_consensus.cc:3037] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:36.057107 27395 raft_consensus.cc:3037] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:36.058955 27395 raft_consensus.cc:2454] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:36.059197 27264 raft_consensus.cc:2454] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:36.059365 27462 leader_election.cc:302] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:36.059687 28464 raft_consensus.cc:2781] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:36.060056 28464 raft_consensus.cc:686] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:36.060329 28464 consensus_queue.cc:237] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.063035 26989 catalog_manager.cc:5484] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:11:36.094780 28492 raft_consensus.cc:482] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:36.095001 28492 raft_consensus.cc:504] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.095623 28492 leader_election.cc:288] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:36.096160 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c619a2235c9248eeb6b2e1d78583d1ba" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:36.096204 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c619a2235c9248eeb6b2e1d78583d1ba" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:36.096436 27264 raft_consensus.cc:2454] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:36.096436 27525 raft_consensus.cc:2454] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:36.096915 27331 leader_election.cc:302] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8b307156e1346bd9eecf655f49924c7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:36.097234 28492 raft_consensus.cc:2781] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:36.097339 28492 raft_consensus.cc:482] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:36.097450 28492 raft_consensus.cc:3037] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:36.099700 28492 raft_consensus.cc:504] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.100180 28492 leader_election.cc:288] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:36.100590 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c619a2235c9248eeb6b2e1d78583d1ba" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:36.100621 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c619a2235c9248eeb6b2e1d78583d1ba" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:36.100792 27525 raft_consensus.cc:3037] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:36.100811 27264 raft_consensus.cc:3037] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:36.102726 27264 raft_consensus.cc:2454] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:36.102886 27525 raft_consensus.cc:2454] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:36.103145 27331 leader_election.cc:302] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8b307156e1346bd9eecf655f49924c7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:36.103641 28492 raft_consensus.cc:2781] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:36.103839 28492 raft_consensus.cc:686] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Becoming Leader. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:11:36.104735 28492 consensus_queue.cc:237] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.117496 26989 catalog_manager.cc:5484] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to e677127814314770b3370764d0ff9d06 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:11:36.166728 28179 raft_consensus.cc:482] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:36.166963 28179 raft_consensus.cc:504] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:36.167953 28179 leader_election.cc:288] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:36.168596 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:36.175995 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:36.176316 27395 raft_consensus.cc:2379] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8b307156e1346bd9eecf655f49924c7 in current term 1: Already voted for candidate 4d603ea5d17441b3bac562bf08ec83fe in this term.
I20240501 20:11:36.176797 27202 leader_election.cc:302] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8b307156e1346bd9eecf655f49924c7; no voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06
I20240501 20:11:36.177145 28179 raft_consensus.cc:3037] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:36.179318 28179 raft_consensus.cc:2726] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:11:36.449280 28464 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:36.454483 28464 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:36.484823 28579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:36.485121 28579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:36.492425 28579 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:36.525679 28546 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240501 20:11:36.538789 28546 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:36.650267 27264 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:36.652964 28492 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:36.700103 27525 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:11:36.701618 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 48803.8us
I20240501 20:11:36.702157 28492 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 266.667 ms
time per row: 0.133333 ms
I20240501 20:11:36.842936 28629 mvcc.cc:205] Tried to move back new op lower bound from 7022978239259844608 to 7022978236847472640. Current Snapshot: MvccSnapshot[applied={T|T < 7022978239259844608}]
I20240501 20:11:36.930982 26993 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:36.941224 27114 tablet_service.cc:1467] Processing CreateTablet for tablet 9721fc9afa4741b78823e293514e6265 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.941720 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9721fc9afa4741b78823e293514e6265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.943147 27505 tablet_service.cc:1467] Processing CreateTablet for tablet d2160f17e62749dbb41405a895516b06 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.943194 27245 tablet_service.cc:1467] Processing CreateTablet for tablet 9721fc9afa4741b78823e293514e6265 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.943439 27115 tablet_service.cc:1467] Processing CreateTablet for tablet d2160f17e62749dbb41405a895516b06 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.943607 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2160f17e62749dbb41405a895516b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.943670 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9721fc9afa4741b78823e293514e6265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.943832 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2160f17e62749dbb41405a895516b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.943969 27113 tablet_service.cc:1467] Processing CreateTablet for tablet e5b9f1a087114c82b010bd10e4f676c1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.944358 27113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b9f1a087114c82b010bd10e4f676c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.949878 27506 tablet_service.cc:1467] Processing CreateTablet for tablet 9721fc9afa4741b78823e293514e6265 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.950330 27506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9721fc9afa4741b78823e293514e6265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.950356 27375 tablet_service.cc:1467] Processing CreateTablet for tablet d2160f17e62749dbb41405a895516b06 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.950800 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2160f17e62749dbb41405a895516b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.954087 28645 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:36.955044 27246 tablet_service.cc:1467] Processing CreateTablet for tablet e5b9f1a087114c82b010bd10e4f676c1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.955250 28646 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:36.955479 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b9f1a087114c82b010bd10e4f676c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.956478 28645 tablet_bootstrap.cc:654] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.958742 28646 tablet_bootstrap.cc:654] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.959177 27504 tablet_service.cc:1467] Processing CreateTablet for tablet e5b9f1a087114c82b010bd10e4f676c1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:36.959687 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b9f1a087114c82b010bd10e4f676c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:36.961889 28644 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:36.964392 28644 tablet_bootstrap.cc:654] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.967711 28646 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:36.968044 28646 ts_tablet_manager.cc:1367] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240501 20:11:36.969362 28646 raft_consensus.cc:348] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.970126 28646 raft_consensus.cc:374] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.970337 28646 raft_consensus.cc:729] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.970748 28646 consensus_queue.cc:260] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.971297 28646 ts_tablet_manager.cc:1398] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:36.972635 28646 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:36.973163 28644 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:36.973367 28644 ts_tablet_manager.cc:1367] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240501 20:11:36.974232 28644 raft_consensus.cc:348] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.974372 28646 tablet_bootstrap.cc:654] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.974537 28644 raft_consensus.cc:374] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.974733 28644 raft_consensus.cc:729] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.975076 28644 consensus_queue.cc:260] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:36.976009 28644 ts_tablet_manager.cc:1398] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:36.979851 28644 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:36.982029 28644 tablet_bootstrap.cc:654] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.984092 28651 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:36.986944 28651 tablet_bootstrap.cc:654] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.989609 28645 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:36.989651 28646 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:36.989831 28646 ts_tablet_manager.cc:1367] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.009s
I20240501 20:11:36.989904 28645 ts_tablet_manager.cc:1367] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.036s user 0.017s sys 0.008s
I20240501 20:11:36.990362 28644 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:36.990587 28644 ts_tablet_manager.cc:1367] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20240501 20:11:36.990831 28646 raft_consensus.cc:348] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.991073 28646 raft_consensus.cc:374] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.991158 28646 raft_consensus.cc:729] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.991132 28645 raft_consensus.cc:348] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:36.991492 28645 raft_consensus.cc:374] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.991467 28646 consensus_queue.cc:260] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:36.991668 28645 raft_consensus.cc:729] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.991634 28644 raft_consensus.cc:348] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:36.991935 28644 raft_consensus.cc:374] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:36.992056 28644 raft_consensus.cc:729] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:36.992060 28645 consensus_queue.cc:260] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:36.992379 28644 consensus_queue.cc:260] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:36.993093 28644 ts_tablet_manager.cc:1398] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:36.993422 28644 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:36.995055 28645 ts_tablet_manager.cc:1398] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:11:36.995476 28645 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:36.996628 28644 tablet_bootstrap.cc:654] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.997794 28645 tablet_bootstrap.cc:654] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:36.999284 28651 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:36.999500 28651 ts_tablet_manager.cc:1367] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20240501 20:11:37.000609 28651 raft_consensus.cc:348] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:37.000895 28651 raft_consensus.cc:374] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:37.001082 28651 raft_consensus.cc:729] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:37.001605 28646 ts_tablet_manager.cc:1398] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 20:11:37.001513 28651 consensus_queue.cc:260] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:37.003343 28651 ts_tablet_manager.cc:1398] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:11:37.004317 28646 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:37.004813 28644 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:37.005023 28644 ts_tablet_manager.cc:1367] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 20:11:37.005834 28644 raft_consensus.cc:348] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:37.006093 28644 raft_consensus.cc:374] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:37.006229 28644 raft_consensus.cc:729] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:37.006482 28646 tablet_bootstrap.cc:654] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:37.006548 28644 consensus_queue.cc:260] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:37.007336 28644 ts_tablet_manager.cc:1398] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:37.011708 28645 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:37.011956 28645 ts_tablet_manager.cc:1367] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20240501 20:11:37.012552 28645 raft_consensus.cc:348] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:37.012732 28645 raft_consensus.cc:374] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:37.012811 28645 raft_consensus.cc:729] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:37.013175 28645 consensus_queue.cc:260] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:37.013780 28645 ts_tablet_manager.cc:1398] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:37.015013 28646 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:37.015161 28646 ts_tablet_manager.cc:1367] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 20:11:37.015916 28646 raft_consensus.cc:348] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:37.016139 28646 raft_consensus.cc:374] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:37.016242 28646 raft_consensus.cc:729] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:37.016487 28646 consensus_queue.cc:260] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:37.016970 28646 ts_tablet_manager.cc:1398] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:37.028105 28179 raft_consensus.cc:482] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:37.028282 28179 raft_consensus.cc:504] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:37.029052 28179 leader_election.cc:288] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:37.029371 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b9f1a087114c82b010bd10e4f676c1" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:37.029613 27134 raft_consensus.cc:2454] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 0.
I20240501 20:11:37.029673 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b9f1a087114c82b010bd10e4f676c1" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:37.029917 27525 raft_consensus.cc:2454] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 0.
I20240501 20:11:37.030283 27201 leader_election.cc:302] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:37.030584 28179 raft_consensus.cc:2781] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:37.030723 28179 raft_consensus.cc:482] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:37.030807 28179 raft_consensus.cc:3037] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.033314 28179 raft_consensus.cc:504] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:37.033892 28179 leader_election.cc:288] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:37.034253 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b9f1a087114c82b010bd10e4f676c1" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:37.034314 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b9f1a087114c82b010bd10e4f676c1" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:37.034493 27134 raft_consensus.cc:3037] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.034505 27525 raft_consensus.cc:3037] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.036432 27134 raft_consensus.cc:2454] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 1.
I20240501 20:11:37.036808 27201 leader_election.cc:302] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:37.037109 28179 raft_consensus.cc:2781] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:37.037292 27525 raft_consensus.cc:2454] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 1.
I20240501 20:11:37.037603 28179 raft_consensus.cc:686] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Becoming Leader. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Running, Role: LEADER
I20240501 20:11:37.037957 28179 consensus_queue.cc:237] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:37.041107 26989 catalog_manager.cc:5484] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8b307156e1346bd9eecf655f49924c7 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:11:37.051123 28492 raft_consensus.cc:482] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:37.051321 28492 raft_consensus.cc:504] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:37.051976 28492 leader_election.cc:288] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:37.052546 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2160f17e62749dbb41405a895516b06" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:37.052531 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2160f17e62749dbb41405a895516b06" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:37.052772 27134 raft_consensus.cc:2454] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:37.052776 27525 raft_consensus.cc:2454] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:37.053169 27331 leader_election.cc:302] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:37.053448 28492 raft_consensus.cc:2781] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:37.053558 28492 raft_consensus.cc:482] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:37.053653 28492 raft_consensus.cc:3037] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.055725 28492 raft_consensus.cc:504] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:37.056282 28492 leader_election.cc:288] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:37.056733 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2160f17e62749dbb41405a895516b06" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:37.056816 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2160f17e62749dbb41405a895516b06" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:37.056912 27525 raft_consensus.cc:3037] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.057036 27134 raft_consensus.cc:3037] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.058903 27134 raft_consensus.cc:2454] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:37.059005 27525 raft_consensus.cc:2454] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:37.059278 27331 leader_election.cc:302] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:37.059576 28492 raft_consensus.cc:2781] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:37.059711 28492 raft_consensus.cc:686] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Becoming Leader. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:11:37.060005 28492 consensus_queue.cc:237] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:37.062821 26989 catalog_manager.cc:5484] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to e677127814314770b3370764d0ff9d06 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:11:37.175419 28464 raft_consensus.cc:482] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:37.175601 28464 raft_consensus.cc:504] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:37.176179 28464 leader_election.cc:288] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:37.176671 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9721fc9afa4741b78823e293514e6265" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:37.176718 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9721fc9afa4741b78823e293514e6265" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:37.176898 27134 raft_consensus.cc:2454] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:37.176939 27264 raft_consensus.cc:2454] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:37.177392 27461 leader_election.cc:302] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:37.177685 28464 raft_consensus.cc:2781] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:37.177793 28464 raft_consensus.cc:482] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:37.177896 28464 raft_consensus.cc:3037] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.179800 28464 raft_consensus.cc:504] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:37.180475 28464 leader_election.cc:288] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:37.180815 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9721fc9afa4741b78823e293514e6265" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:37.181016 27134 raft_consensus.cc:3037] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.181061 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9721fc9afa4741b78823e293514e6265" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:37.181286 27264 raft_consensus.cc:3037] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.182955 27134 raft_consensus.cc:2454] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:37.183295 27264 raft_consensus.cc:2454] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:37.183352 27461 leader_election.cc:302] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:37.183615 28464 raft_consensus.cc:2781] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:37.183830 28464 raft_consensus.cc:686] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:37.184101 28464 consensus_queue.cc:237] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:11:37.187248 26989 catalog_manager.cc:5484] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:11:37.449477 28179 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:37.455443 28657 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:37.579731 28492 consensus_queue.cc:1035] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:37.596225 28492 consensus_queue.cc:1035] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:37.617648 28661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:37.618028 28661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:37.626437 28661 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 20:11:37.736050 28576 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:37.736729 28576 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 269.403 ms
time per row: 0.134702 ms
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:11:38.997188 26989 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:39.006824 27246 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc5794a67374cc388da2edc47cbb113 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.007309 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc5794a67374cc388da2edc47cbb113. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.007927 27504 tablet_service.cc:1467] Processing CreateTablet for tablet e7f03e02d13f44d188e5eb2774771a0d (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.008319 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f03e02d13f44d188e5eb2774771a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.008621 27374 tablet_service.cc:1467] Processing CreateTablet for tablet 944d354124664bc0870f251d8a0831c5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.009068 27374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d354124664bc0870f251d8a0831c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.011117 27376 tablet_service.cc:1467] Processing CreateTablet for tablet e7f03e02d13f44d188e5eb2774771a0d (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.011516 27376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f03e02d13f44d188e5eb2774771a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.014957 27505 tablet_service.cc:1467] Processing CreateTablet for tablet 944d354124664bc0870f251d8a0831c5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.015369 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d354124664bc0870f251d8a0831c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.015334 27245 tablet_service.cc:1467] Processing CreateTablet for tablet 944d354124664bc0870f251d8a0831c5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.015473 27375 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc5794a67374cc388da2edc47cbb113 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.015802 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d354124664bc0870f251d8a0831c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.016337 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc5794a67374cc388da2edc47cbb113. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.017283 28736 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:39.026453 28737 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:39.027158 28736 tablet_bootstrap.cc:654] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.028643 28737 tablet_bootstrap.cc:654] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.032334 27115 tablet_service.cc:1467] Processing CreateTablet for tablet e7f03e02d13f44d188e5eb2774771a0d (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.032337 27114 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc5794a67374cc388da2edc47cbb113 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.032824 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc5794a67374cc388da2edc47cbb113. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.035382 28735 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:39.037822 28735 tablet_bootstrap.cc:654] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.038873 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f03e02d13f44d188e5eb2774771a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.045933 28741 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:39.047691 28741 tablet_bootstrap.cc:654] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.049218 28737 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:39.049408 28737 ts_tablet_manager.cc:1367] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240501 20:11:39.049729 28736 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:39.050005 28736 ts_tablet_manager.cc:1367] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.015s
I20240501 20:11:39.050248 28737 raft_consensus.cc:348] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.050566 28737 raft_consensus.cc:374] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.050725 28737 raft_consensus.cc:729] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.050752 28736 raft_consensus.cc:348] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.051054 28736 raft_consensus.cc:374] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.051024 28737 consensus_queue.cc:260] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.051273 28736 raft_consensus.cc:729] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.051332 28735 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:39.051476 28735 ts_tablet_manager.cc:1367] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240501 20:11:39.051612 28737 ts_tablet_manager.cc:1398] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:39.051645 28736 consensus_queue.cc:260] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.051940 28737 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:39.052244 28735 raft_consensus.cc:348] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.052453 28735 raft_consensus.cc:374] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.052563 28735 raft_consensus.cc:729] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.052779 28735 consensus_queue.cc:260] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.053333 28735 ts_tablet_manager.cc:1398] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:39.053822 28737 tablet_bootstrap.cc:654] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.055651 28736 ts_tablet_manager.cc:1398] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:11:39.058089 28736 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:39.058100 28741 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:39.058240 28741 ts_tablet_manager.cc:1367] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 20:11:39.059005 28741 raft_consensus.cc:348] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.059247 28741 raft_consensus.cc:374] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.059379 28741 raft_consensus.cc:729] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.059631 28741 consensus_queue.cc:260] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.059795 28736 tablet_bootstrap.cc:654] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.060266 28741 ts_tablet_manager.cc:1398] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:39.060578 28741 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:39.061952 28741 tablet_bootstrap.cc:654] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.063617 28741 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:39.063825 28741 ts_tablet_manager.cc:1367] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:39.064548 28741 raft_consensus.cc:348] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.064754 28741 raft_consensus.cc:374] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.064829 28741 raft_consensus.cc:729] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.065066 28741 consensus_queue.cc:260] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.065460 28741 ts_tablet_manager.cc:1398] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:39.065821 28735 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:39.069629 28464 raft_consensus.cc:482] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:39.069842 28464 raft_consensus.cc:504] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.071440 28464 leader_election.cc:288] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:39.071429 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:39.071688 27134 raft_consensus.cc:2454] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:39.072315 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:39.072445 27461 leader_election.cc:302] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:39.072800 28464 raft_consensus.cc:2781] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:39.072950 28464 raft_consensus.cc:482] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:39.073067 28464 raft_consensus.cc:3037] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:11:39.075181 27462 leader_election.cc:341] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:39.075609 28735 tablet_bootstrap.cc:654] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.078068 28737 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:39.078001 28464 raft_consensus.cc:504] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.078392 28736 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:39.078392 28737 ts_tablet_manager.cc:1367] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.015s
I20240501 20:11:39.078572 28736 ts_tablet_manager.cc:1367] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.010s
I20240501 20:11:39.078950 28464 leader_election.cc:288] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:39.079362 28736 raft_consensus.cc:348] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.079586 28736 raft_consensus.cc:374] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.079649 28736 raft_consensus.cc:729] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.079877 28736 consensus_queue.cc:260] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.080003 28737 raft_consensus.cc:348] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.080269 28737 raft_consensus.cc:374] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.080334 28737 raft_consensus.cc:729] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.080291 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:39.080511 27134 raft_consensus.cc:3037] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.080593 28736 ts_tablet_manager.cc:1398] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:39.080579 28737 consensus_queue.cc:260] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.080881 28736 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:39.081223 28737 ts_tablet_manager.cc:1398] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:39.082448 28736 tablet_bootstrap.cc:654] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.082823 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
W20240501 20:11:39.085796 27462 leader_election.cc:341] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:39.087953 27134 raft_consensus.cc:2454] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:39.089092 27461 leader_election.cc:302] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters: e677127814314770b3370764d0ff9d06
I20240501 20:11:39.089164 28735 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:39.089320 28735 ts_tablet_manager.cc:1367] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.002s
I20240501 20:11:39.089531 28464 raft_consensus.cc:2781] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:39.089706 28736 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:39.089854 28736 ts_tablet_manager.cc:1367] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 20:11:39.090025 28735 raft_consensus.cc:348] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.090212 28735 raft_consensus.cc:374] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.090291 28735 raft_consensus.cc:729] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.090538 28735 consensus_queue.cc:260] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.090605 28736 raft_consensus.cc:348] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.090907 28736 raft_consensus.cc:374] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.091055 28736 raft_consensus.cc:729] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.091168 28735 ts_tablet_manager.cc:1398] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:39.091323 28736 consensus_queue.cc:260] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.091948 28736 ts_tablet_manager.cc:1398] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:39.098548 28464 raft_consensus.cc:686] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:39.098870 28464 consensus_queue.cc:237] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.101493 26991 catalog_manager.cc:5484] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:11:39.158380 28464 raft_consensus.cc:482] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:39.158561 28464 raft_consensus.cc:504] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.159143 28464 leader_election.cc:288] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:39.159613 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d354124664bc0870f251d8a0831c5" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:39.159654 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d354124664bc0870f251d8a0831c5" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:39.159844 27265 raft_consensus.cc:2454] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:39.159855 27394 raft_consensus.cc:2454] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:39.160279 27462 leader_election.cc:302] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:39.160560 28464 raft_consensus.cc:2781] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:39.160645 28464 raft_consensus.cc:482] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:39.160732 28464 raft_consensus.cc:3037] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.162645 28464 raft_consensus.cc:504] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.163205 28464 leader_election.cc:288] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:39.163623 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d354124664bc0870f251d8a0831c5" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:39.163673 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d354124664bc0870f251d8a0831c5" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:39.163805 27394 raft_consensus.cc:3037] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.163856 27265 raft_consensus.cc:3037] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.165696 27265 raft_consensus.cc:2454] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:39.165702 27394 raft_consensus.cc:2454] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:39.166092 27461 leader_election.cc:302] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:39.166404 28464 raft_consensus.cc:2781] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:39.166605 28464 raft_consensus.cc:686] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:39.166905 28464 consensus_queue.cc:237] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.169642 26991 catalog_manager.cc:5484] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:11:39.265038 28667 raft_consensus.cc:482] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:39.265203 28667 raft_consensus.cc:504] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.265805 28667 leader_election.cc:288] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:39.266180 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc5794a67374cc388da2edc47cbb113" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:39.266263 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc5794a67374cc388da2edc47cbb113" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:39.266391 27265 raft_consensus.cc:2454] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:39.266566 27394 raft_consensus.cc:2454] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:39.266813 27070 leader_election.cc:302] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:39.267062 28667 raft_consensus.cc:2781] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:39.267184 28667 raft_consensus.cc:482] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:39.267279 28667 raft_consensus.cc:3037] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.269182 28667 raft_consensus.cc:504] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.269759 28667 leader_election.cc:288] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:39.270123 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc5794a67374cc388da2edc47cbb113" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:39.270323 27265 raft_consensus.cc:3037] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.270269 27394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc5794a67374cc388da2edc47cbb113" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:39.270478 27394 raft_consensus.cc:3037] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.273161 27265 raft_consensus.cc:2454] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:39.273197 27394 raft_consensus.cc:2454] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:39.273574 27070 leader_election.cc:302] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:39.273917 28667 raft_consensus.cc:2781] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:39.274356 28667 raft_consensus.cc:686] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:39.274825 28667 consensus_queue.cc:237] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:39.278411 26991 catalog_manager.cc:5484] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
I20240501 20:11:39.459153 28672 raft_consensus.cc:482] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:39.459343 28672 raft_consensus.cc:504] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:39.459987 28672 leader_election.cc:288] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:39.460454 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:39.460536 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:39.460769 27134 raft_consensus.cc:2379] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e677127814314770b3370764d0ff9d06 in current term 1: Already voted for candidate 524f5e87a3ce40378bde76e667a4ece7 in this term.
I20240501 20:11:39.461244 27330 leader_election.cc:302] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e677127814314770b3370764d0ff9d06; no voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7
I20240501 20:11:39.461810 28672 raft_consensus.cc:3037] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.463900 28672 raft_consensus.cc:2726] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:11:39.522343 28464 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:39.537849 28748 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:39.668318 28748 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:39.673317 28464 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:39.673360 28755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:39.673782 28755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:39.682997 28755 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 20:11:39.785504 28752 consensus_queue.cc:1035] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:39.787215 28752 consensus_queue.cc:1035] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 245.365 ms
time per row: 0.122682 ms
I20240501 20:11:40.079496 26987 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:40.088469 27504 tablet_service.cc:1467] Processing CreateTablet for tablet 5272c34458af43bebb5ba5baefac7137 (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.088644 27114 tablet_service.cc:1467] Processing CreateTablet for tablet 5272c34458af43bebb5ba5baefac7137 (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.088969 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5272c34458af43bebb5ba5baefac7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.088907 27374 tablet_service.cc:1467] Processing CreateTablet for tablet ebcdab26b4ed45f596282d4805d1bb1f (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.089037 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5272c34458af43bebb5ba5baefac7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.089293 27374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcdab26b4ed45f596282d4805d1bb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.095067 27246 tablet_service.cc:1467] Processing CreateTablet for tablet 5272c34458af43bebb5ba5baefac7137 (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.095579 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5272c34458af43bebb5ba5baefac7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.095773 28824 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:40.097328 27376 tablet_service.cc:1467] Processing CreateTablet for tablet 00e3777032eb4d9c80f47855e5f9f87a (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.097692 27376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e3777032eb4d9c80f47855e5f9f87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.098322 27115 tablet_service.cc:1467] Processing CreateTablet for tablet ebcdab26b4ed45f596282d4805d1bb1f (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.098727 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcdab26b4ed45f596282d4805d1bb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.098685 27504 tablet_service.cc:1467] Processing CreateTablet for tablet 00e3777032eb4d9c80f47855e5f9f87a (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.099035 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e3777032eb4d9c80f47855e5f9f87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.099485 28824 tablet_bootstrap.cc:654] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.101400 27113 tablet_service.cc:1467] Processing CreateTablet for tablet 00e3777032eb4d9c80f47855e5f9f87a (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.101856 27113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e3777032eb4d9c80f47855e5f9f87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.103053 27245 tablet_service.cc:1467] Processing CreateTablet for tablet ebcdab26b4ed45f596282d4805d1bb1f (DEFAULT_TABLE table=rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.103549 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcdab26b4ed45f596282d4805d1bb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.120777 28827 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:40.122994 28827 tablet_bootstrap.cc:654] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.123590 28825 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:40.126587 28825 tablet_bootstrap.cc:654] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.128849 28827 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:40.129031 28827 ts_tablet_manager.cc:1367] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240501 20:11:40.130039 28827 raft_consensus.cc:348] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.130298 28827 raft_consensus.cc:374] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.130383 28827 raft_consensus.cc:729] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.130607 28827 consensus_queue.cc:260] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.131398 28827 ts_tablet_manager.cc:1398] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:11:40.132359 28827 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:40.133880 28827 tablet_bootstrap.cc:654] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.140976 28827 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:40.141484 28827 ts_tablet_manager.cc:1367] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240501 20:11:40.142920 28827 raft_consensus.cc:348] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.143273 28827 raft_consensus.cc:374] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.143425 28827 raft_consensus.cc:729] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.143563 28823 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:40.144222 28824 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:40.144383 28824 ts_tablet_manager.cc:1367] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.020s
I20240501 20:11:40.145022 28824 raft_consensus.cc:348] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.145184 28824 raft_consensus.cc:374] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.145290 28824 raft_consensus.cc:729] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.145529 28824 consensus_queue.cc:260] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.145956 28823 tablet_bootstrap.cc:654] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.146194 28824 ts_tablet_manager.cc:1398] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:11:40.146466 28824 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:40.148102 28824 tablet_bootstrap.cc:654] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.149093 28827 consensus_queue.cc:260] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.152884 28827 ts_tablet_manager.cc:1398] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:11:40.153869 28825 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:40.153963 28824 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:40.154091 28825 ts_tablet_manager.cc:1367] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.013s
I20240501 20:11:40.154160 28824 ts_tablet_manager.cc:1367] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:11:40.154317 28823 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:40.154604 28823 ts_tablet_manager.cc:1367] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240501 20:11:40.154955 28824 raft_consensus.cc:348] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.155027 28825 raft_consensus.cc:348] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.155238 28824 raft_consensus.cc:374] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.155272 28825 raft_consensus.cc:374] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.155361 28824 raft_consensus.cc:729] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.155414 28825 raft_consensus.cc:729] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.155550 28823 raft_consensus.cc:348] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.155715 28825 consensus_queue.cc:260] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.155714 28824 consensus_queue.cc:260] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.155880 28823 raft_consensus.cc:374] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.156028 28823 raft_consensus.cc:729] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.156347 28824 ts_tablet_manager.cc:1398] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:40.156318 28823 consensus_queue.cc:260] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.157069 28825 ts_tablet_manager.cc:1398] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:40.157668 28823 ts_tablet_manager.cc:1398] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 20:11:40.158010 28823 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:40.159118 28825 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:40.160391 28823 tablet_bootstrap.cc:654] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.160758 28825 tablet_bootstrap.cc:654] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.163941 28672 raft_consensus.cc:482] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:40.164147 28672 raft_consensus.cc:504] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.164945 28672 leader_election.cc:288] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:40.165896 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcdab26b4ed45f596282d4805d1bb1f" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:40.166173 27264 raft_consensus.cc:2454] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:40.166246 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcdab26b4ed45f596282d4805d1bb1f" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:40.166941 27331 leader_election.cc:302] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8b307156e1346bd9eecf655f49924c7, e677127814314770b3370764d0ff9d06; no voters:
W20240501 20:11:40.167415 27331 leader_election.cc:341] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:40.167944 28672 raft_consensus.cc:2781] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:40.168057 28672 raft_consensus.cc:482] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:40.168166 28672 raft_consensus.cc:3037] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.168982 28823 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:40.169298 28823 ts_tablet_manager.cc:1367] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 20:11:40.170078 28823 raft_consensus.cc:348] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.170285 28823 raft_consensus.cc:374] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.170353 28823 raft_consensus.cc:729] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.170616 28823 consensus_queue.cc:260] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.171011 28672 raft_consensus.cc:504] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.172449 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcdab26b4ed45f596282d4805d1bb1f" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:40.172684 27264 raft_consensus.cc:3037] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.174773 28823 ts_tablet_manager.cc:1398] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 20:11:40.175626 27264 raft_consensus.cc:2454] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:40.176116 27331 leader_election.cc:302] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8b307156e1346bd9eecf655f49924c7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:40.176630 28672 leader_election.cc:288] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:40.176795 28672 raft_consensus.cc:2781] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:40.176957 28672 raft_consensus.cc:686] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Becoming Leader. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:11:40.177176 28672 consensus_queue.cc:237] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.178076 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcdab26b4ed45f596282d4805d1bb1f" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
W20240501 20:11:40.178783 27331 leader_election.cc:341] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:40.179132 28825 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:40.179312 28825 ts_tablet_manager.cc:1367] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240501 20:11:40.179975 28825 raft_consensus.cc:348] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.180146 28825 raft_consensus.cc:374] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.180233 28825 raft_consensus.cc:729] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.180498 28825 consensus_queue.cc:260] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.181043 28825 ts_tablet_manager.cc:1398] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:40.181373 28825 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:40.181910 26987 catalog_manager.cc:5484] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to e677127814314770b3370764d0ff9d06 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
I20240501 20:11:40.182886 28825 tablet_bootstrap.cc:654] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.184646 28825 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:40.184849 28825 ts_tablet_manager.cc:1367] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:40.185667 28825 raft_consensus.cc:348] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.185875 28825 raft_consensus.cc:374] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.186096 28825 raft_consensus.cc:729] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.186324 28825 consensus_queue.cc:260] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.186889 28825 ts_tablet_manager.cc:1398] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:40.273548 28813 raft_consensus.cc:482] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:40.273748 28813 raft_consensus.cc:504] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.274703 28813 leader_election.cc:288] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:40.275375 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e3777032eb4d9c80f47855e5f9f87a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:40.275627 27396 raft_consensus.cc:2454] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:40.275557 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e3777032eb4d9c80f47855e5f9f87a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:11:40.275837 27134 raft_consensus.cc:2454] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:40.276332 27462 leader_election.cc:302] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:40.276708 28813 raft_consensus.cc:2781] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:40.276837 28813 raft_consensus.cc:482] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:40.276938 28813 raft_consensus.cc:3037] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.279433 28813 raft_consensus.cc:504] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.280043 28813 leader_election.cc:288] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:40.280453 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e3777032eb4d9c80f47855e5f9f87a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:40.280694 27396 raft_consensus.cc:3037] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.280630 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e3777032eb4d9c80f47855e5f9f87a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:11:40.280869 27134 raft_consensus.cc:3037] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.283636 27396 raft_consensus.cc:2454] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:40.283674 27134 raft_consensus.cc:2454] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:40.284132 27461 leader_election.cc:302] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:40.284612 28813 raft_consensus.cc:2781] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:40.284755 28813 raft_consensus.cc:686] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:40.285056 28813 consensus_queue.cc:237] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.288905 26987 catalog_manager.cc:5484] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
I20240501 20:11:40.346325 28752 raft_consensus.cc:482] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:40.346489 28752 raft_consensus.cc:504] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.347227 28752 leader_election.cc:288] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:40.347750 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcdab26b4ed45f596282d4805d1bb1f" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:40.347800 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcdab26b4ed45f596282d4805d1bb1f" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:40.348129 27264 raft_consensus.cc:2379] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d603ea5d17441b3bac562bf08ec83fe in current term 1: Already voted for candidate e677127814314770b3370764d0ff9d06 in this term.
I20240501 20:11:40.348569 27070 leader_election.cc:302] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe; no voters: d8b307156e1346bd9eecf655f49924c7, e677127814314770b3370764d0ff9d06
I20240501 20:11:40.348893 28752 raft_consensus.cc:3037] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.351020 28752 raft_consensus.cc:2726] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:11:40.441787 28752 raft_consensus.cc:482] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:40.442004 28752 raft_consensus.cc:504] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.442657 28752 leader_election.cc:288] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:40.443250 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5272c34458af43bebb5ba5baefac7137" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:40.443228 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5272c34458af43bebb5ba5baefac7137" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:40.443476 27525 raft_consensus.cc:2454] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:40.443498 27264 raft_consensus.cc:2454] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:40.443869 27069 leader_election.cc:302] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:40.444128 28752 raft_consensus.cc:2781] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:40.444258 28752 raft_consensus.cc:482] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:40.444392 28752 raft_consensus.cc:3037] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.446403 28752 raft_consensus.cc:504] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.447041 28752 leader_election.cc:288] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:40.447362 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5272c34458af43bebb5ba5baefac7137" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:40.447520 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5272c34458af43bebb5ba5baefac7137" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:40.447629 27264 raft_consensus.cc:3037] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.447733 27525 raft_consensus.cc:3037] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.450570 27525 raft_consensus.cc:2454] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:40.450567 27264 raft_consensus.cc:2454] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:40.451054 27070 leader_election.cc:302] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:40.451370 28752 raft_consensus.cc:2781] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:40.451530 28752 raft_consensus.cc:686] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:40.451792 28752 consensus_queue.cc:237] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:11:40.454839 26987 catalog_manager.cc:5484] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
I20240501 20:11:40.612154 28672 consensus_queue.cc:1035] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:40.617185 28672 consensus_queue.cc:1035] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:40.865196 28752 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:40.871060 28752 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:40.893680 28809 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:40.898962 28813 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:40.901594 28841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:40.902035 28841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:40.910750 28841 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 251.552 ms
time per row: 0.125776 ms
I20240501 20:11:41.324535 26994 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:41.333607 27245 tablet_service.cc:1467] Processing CreateTablet for tablet 0c836bb145a54160966c475e1490da38 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.333808 27504 tablet_service.cc:1467] Processing CreateTablet for tablet 0c836bb145a54160966c475e1490da38 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.334180 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c836bb145a54160966c475e1490da38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.334268 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c836bb145a54160966c475e1490da38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.334750 27376 tablet_service.cc:1467] Processing CreateTablet for tablet 57b4bd50bc8f4d50957bd0280224c420 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.335124 27376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b4bd50bc8f4d50957bd0280224c420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.339016 27374 tablet_service.cc:1467] Processing CreateTablet for tablet 0c836bb145a54160966c475e1490da38 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.339383 27374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c836bb145a54160966c475e1490da38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.343156 27505 tablet_service.cc:1467] Processing CreateTablet for tablet 57b4bd50bc8f4d50957bd0280224c420 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.343662 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b4bd50bc8f4d50957bd0280224c420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.347371 28911 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:41.347892 27246 tablet_service.cc:1467] Processing CreateTablet for tablet 57b4bd50bc8f4d50957bd0280224c420 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.348013 27244 tablet_service.cc:1467] Processing CreateTablet for tablet c2432c8e8e8847fa977055be919e79d6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.348407 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b4bd50bc8f4d50957bd0280224c420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.348677 27244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2432c8e8e8847fa977055be919e79d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.349112 27114 tablet_service.cc:1467] Processing CreateTablet for tablet c2432c8e8e8847fa977055be919e79d6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.349516 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2432c8e8e8847fa977055be919e79d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.355861 27504 tablet_service.cc:1467] Processing CreateTablet for tablet c2432c8e8e8847fa977055be919e79d6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.356494 28912 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:11:41.356752 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2432c8e8e8847fa977055be919e79d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.357720 28911 tablet_bootstrap.cc:654] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.358392 28912 tablet_bootstrap.cc:654] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.358873 28910 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:41.361052 28910 tablet_bootstrap.cc:654] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.368371 28909 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:41.370977 28909 tablet_bootstrap.cc:654] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.372448 28912 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe: No bootstrap required, opened a new log
I20240501 20:11:41.372651 28912 ts_tablet_manager.cc:1367] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20240501 20:11:41.374176 28912 raft_consensus.cc:348] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.374562 28912 raft_consensus.cc:374] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.374753 28912 raft_consensus.cc:729] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.375164 28912 consensus_queue.cc:260] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.380069 28912 ts_tablet_manager.cc:1398] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:11:41.384555 28909 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:41.384646 28910 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:41.384714 28909 ts_tablet_manager.cc:1367] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20240501 20:11:41.384771 28911 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:41.384822 28910 ts_tablet_manager.cc:1367] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.011s
I20240501 20:11:41.384975 28911 ts_tablet_manager.cc:1367] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.011s
I20240501 20:11:41.385531 28910 raft_consensus.cc:348] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.385545 28909 raft_consensus.cc:348] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.385699 28910 raft_consensus.cc:374] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.385768 28909 raft_consensus.cc:374] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.385787 28910 raft_consensus.cc:729] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.385717 28911 raft_consensus.cc:348] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.385874 28909 raft_consensus.cc:729] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.385969 28911 raft_consensus.cc:374] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.386103 28911 raft_consensus.cc:729] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.386060 28910 consensus_queue.cc:260] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.386166 28909 consensus_queue.cc:260] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.386376 28911 consensus_queue.cc:260] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.387367 28910 ts_tablet_manager.cc:1398] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:11:41.387511 28911 ts_tablet_manager.cc:1398] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:41.387706 28910 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:41.387856 28911 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:41.389078 28909 ts_tablet_manager.cc:1398] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:11:41.389379 28910 tablet_bootstrap.cc:654] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.389423 28909 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:11:41.389520 28911 tablet_bootstrap.cc:654] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.396620 28910 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:41.396840 28910 ts_tablet_manager.cc:1367] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240501 20:11:41.397330 28909 tablet_bootstrap.cc:654] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.397557 28910 raft_consensus.cc:348] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.397794 28910 raft_consensus.cc:374] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.397888 28910 raft_consensus.cc:729] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.398154 28910 consensus_queue.cc:260] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.398775 28911 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:41.398802 28910 ts_tablet_manager.cc:1398] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:41.398941 28911 ts_tablet_manager.cc:1367] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 20:11:41.399091 28910 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:11:41.399606 28911 raft_consensus.cc:348] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.399816 28911 raft_consensus.cc:374] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.399899 28911 raft_consensus.cc:729] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.400151 28911 consensus_queue.cc:260] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.400775 28911 ts_tablet_manager.cc:1398] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:41.400843 28910 tablet_bootstrap.cc:654] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.401216 28911 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:11:41.401764 28909 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: No bootstrap required, opened a new log
I20240501 20:11:41.402992 28911 tablet_bootstrap.cc:654] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.403419 28909 ts_tablet_manager.cc:1367] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 20:11:41.407478 28909 raft_consensus.cc:348] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.409859 28909 raft_consensus.cc:374] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.409961 28909 raft_consensus.cc:729] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.410228 28909 consensus_queue.cc:260] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.410933 28909 ts_tablet_manager.cc:1398] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:11:41.411103 28911 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: No bootstrap required, opened a new log
I20240501 20:11:41.411293 28910 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: No bootstrap required, opened a new log
I20240501 20:11:41.411286 28911 ts_tablet_manager.cc:1367] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:11:41.411432 28910 ts_tablet_manager.cc:1367] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240501 20:11:41.412026 28911 raft_consensus.cc:348] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.412253 28911 raft_consensus.cc:374] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.412137 28910 raft_consensus.cc:348] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.412354 28910 raft_consensus.cc:374] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.412374 28911 raft_consensus.cc:729] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.412480 28910 raft_consensus.cc:729] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.412639 28911 consensus_queue.cc:260] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.412755 28910 consensus_queue.cc:260] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.413197 28911 ts_tablet_manager.cc:1398] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:41.414280 28910 ts_tablet_manager.cc:1398] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:11:41.434228 28903 raft_consensus.cc:482] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:41.434370 28903 raft_consensus.cc:504] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.435096 28903 leader_election.cc:288] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:41.435524 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b4bd50bc8f4d50957bd0280224c420" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:41.435561 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b4bd50bc8f4d50957bd0280224c420" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:41.435742 27264 raft_consensus.cc:2454] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:41.435768 27525 raft_consensus.cc:2454] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 0.
I20240501 20:11:41.436226 27330 leader_election.cc:302] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:41.436570 28903 raft_consensus.cc:2781] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:41.436688 28903 raft_consensus.cc:482] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:41.436784 28903 raft_consensus.cc:3037] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.438725 28903 raft_consensus.cc:504] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.439273 28903 leader_election.cc:288] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:41.439672 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b4bd50bc8f4d50957bd0280224c420" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:41.439682 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b4bd50bc8f4d50957bd0280224c420" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:41.439847 27264 raft_consensus.cc:3037] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.439846 27525 raft_consensus.cc:3037] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.442056 27525 raft_consensus.cc:2454] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:41.442248 27264 raft_consensus.cc:2454] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 1.
I20240501 20:11:41.442497 27330 leader_election.cc:302] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:41.442781 28903 raft_consensus.cc:2781] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:41.442920 28903 raft_consensus.cc:686] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Becoming Leader. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:11:41.443235 28903 consensus_queue.cc:237] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.445894 26994 catalog_manager.cc:5484] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to e677127814314770b3370764d0ff9d06 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:11:41.577174 28809 raft_consensus.cc:482] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:41.577361 28809 raft_consensus.cc:504] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.578606 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c836bb145a54160966c475e1490da38" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06" is_pre_election: true
I20240501 20:11:41.578644 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c836bb145a54160966c475e1490da38" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:41.578866 27396 raft_consensus.cc:2454] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:41.578943 27264 raft_consensus.cc:2454] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 0.
I20240501 20:11:41.579360 27461 leader_election.cc:302] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:41.579594 28809 leader_election.cc:288] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:41.579659 28813 raft_consensus.cc:2781] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:41.579770 28813 raft_consensus.cc:482] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:41.579869 28813 raft_consensus.cc:3037] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.582705 28813 raft_consensus.cc:504] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.583346 28813 leader_election.cc:288] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:41.583698 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c836bb145a54160966c475e1490da38" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:41.583762 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c836bb145a54160966c475e1490da38" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:11:41.583936 27396 raft_consensus.cc:3037] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.584045 27264 raft_consensus.cc:3037] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.586805 27396 raft_consensus.cc:2454] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:41.586855 27264 raft_consensus.cc:2454] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
I20240501 20:11:41.587205 27462 leader_election.cc:302] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:11:41.587462 28813 raft_consensus.cc:2781] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:41.587630 28813 raft_consensus.cc:686] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:11:41.587875 28813 consensus_queue.cc:237] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:11:41.590452 26994 catalog_manager.cc:5484] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:11:41.623091 28789 raft_consensus.cc:482] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:41.623288 28789 raft_consensus.cc:504] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.624080 28789 leader_election.cc:288] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:41.624423 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2432c8e8e8847fa977055be919e79d6" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7" is_pre_election: true
I20240501 20:11:41.624661 27264 raft_consensus.cc:2454] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:41.624598 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2432c8e8e8847fa977055be919e79d6" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:11:41.624822 27525 raft_consensus.cc:2454] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 0.
I20240501 20:11:41.625059 27070 leader_election.cc:302] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:11:41.625345 28789 raft_consensus.cc:2781] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:41.625455 28789 raft_consensus.cc:482] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:41.625520 28789 raft_consensus.cc:3037] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.627462 28789 raft_consensus.cc:504] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.628029 28789 leader_election.cc:288] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:41.628561 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2432c8e8e8847fa977055be919e79d6" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:11:41.628561 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2432c8e8e8847fa977055be919e79d6" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:11:41.628751 27525 raft_consensus.cc:3037] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.628770 27265 raft_consensus.cc:3037] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.631544 27265 raft_consensus.cc:2454] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:41.631574 27525 raft_consensus.cc:2454] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 1.
I20240501 20:11:41.631970 27069 leader_election.cc:302] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:11:41.632257 28789 raft_consensus.cc:2781] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:41.632452 28789 raft_consensus.cc:686] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:11:41.632725 28789 consensus_queue.cc:237] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:11:41.635653 26994 catalog_manager.cc:5484] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:11:41.830112 28903 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:41.840557 28929 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:41.966394 28813 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:41.972991 28809 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:42.010340 28925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:42.010712 28925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:42.018846 28925 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 20:11:42.092372 28752 consensus_queue.cc:1035] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:42.113152 28789 consensus_queue.cc:1035] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 314.71 ms
time per row: 0.157355 ms
I20240501 20:11:42.481302 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:40723
--local_ip_for_outbound_sockets=127.14.106.197
--tserver_master_addrs=127.14.106.254:34283
--webserver_port=37495
--webserver_interface=127.14.106.197
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:11:42.858234 28988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:42.858568 28988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:42.858760 28988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:42.866803 28988 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:42.866916 28988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:42.867105 28988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:11:42.876044 28988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:40723
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=37495
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:42.877398 28988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:42.879421 28988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:42.892825 28996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:42.893525 28993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:42.893940 28994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:42.894466 28988 server_base.cc:1034] running on GCE node
I20240501 20:11:42.895126 28988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:42.896102 28988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:42.897298 28988 hybrid_clock.cc:648] HybridClock initialized: now 1714594302897251 us; error 40 us; skew 500 ppm
I20240501 20:11:42.897617 28988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:42.899914 28988 webserver.cc:460] Webserver started at http://127.14.106.197:37495/ using document root <none> and password file <none>
I20240501 20:11:42.900701 28988 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:42.900877 28988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:42.906702 28988 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 20:11:42.909353 29002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:42.910607 28988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:42.910799 28988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "565cade0058540a4aa18d7c7a25affbe"
format_stamp: "Formatted at 2024-05-01 20:11:22 on dist-test-slave-pg4x"
I20240501 20:11:42.911571 28988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:42.940737 28988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:42.941792 28988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:42.942139 28988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:42.943234 28988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:42.944729 28988 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:42.944808 28988 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:42.944936 28988 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:42.944993 28988 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:42.982282 28988 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:40723
I20240501 20:11:42.982343 29114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:40723 every 8 connection(s)
I20240501 20:11:42.984081 28988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:11:42.989313 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 28988
I20240501 20:11:42.990080 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.198:46843
--local_ip_for_outbound_sockets=127.14.106.198
--tserver_master_addrs=127.14.106.254:34283
--webserver_port=33949
--webserver_interface=127.14.106.198
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:43.003840 29115 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:43.004149 29115 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:43.005059 29115 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:43.006770 26993 ts_manager.cc:194] Re-registered known tserver with Master: 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
I20240501 20:11:43.007611 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:48473
W20240501 20:11:43.353319 29119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:43.353623 29119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:43.353765 29119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:43.361328 29119 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:43.361478 29119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:43.361636 29119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.198
I20240501 20:11:43.370836 29119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.198:46843
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.106.198
--webserver_port=33949
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.198
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:43.372110 29119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:43.373813 29119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:43.386561 29126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:43.387126 29127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:43.387622 29129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:43.388078 29119 server_base.cc:1034] running on GCE node
I20240501 20:11:43.388631 29119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:43.389655 29119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:43.390848 29119 hybrid_clock.cc:648] HybridClock initialized: now 1714594303390787 us; error 56 us; skew 500 ppm
I20240501 20:11:43.391151 29119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:43.393244 29119 webserver.cc:460] Webserver started at http://127.14.106.198:33949/ using document root <none> and password file <none>
I20240501 20:11:43.393920 29119 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:43.394052 29119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:43.399206 29119 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:43.402135 29135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.403564 29119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:43.403708 29119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d56cbcb06b9e418cb617de6a34b946b0"
format_stamp: "Formatted at 2024-05-01 20:11:23 on dist-test-slave-pg4x"
I20240501 20:11:43.404676 29119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:43.444244 29119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:43.445015 29119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:43.445312 29119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:43.446358 29119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:43.447906 29119 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:43.447981 29119 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.448066 29119 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:43.448112 29119 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.479735 29119 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.198:46843
I20240501 20:11:43.479789 29247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.198:46843 every 8 connection(s)
I20240501 20:11:43.481158 29119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:11:43.486744 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 29119
I20240501 20:11:43.487500 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.199:32969
--local_ip_for_outbound_sockets=127.14.106.199
--tserver_master_addrs=127.14.106.254:34283
--webserver_port=35385
--webserver_interface=127.14.106.199
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:43.497915 29248 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:43.498317 29248 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:43.499444 29248 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:43.501171 26993 ts_manager.cc:194] Re-registered known tserver with Master: d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843)
I20240501 20:11:43.502276 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.198:56089
W20240501 20:11:43.849352 29253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:43.849647 29253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:43.849766 29253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:43.857983 29253 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:43.858101 29253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:43.858220 29253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.199
I20240501 20:11:43.867633 29253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.199:32969
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.106.199
--webserver_port=35385
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.199
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:43.868954 29253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:43.870608 29253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:43.883265 29260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:43.883683 29262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:43.884313 29259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:43.884727 29253 server_base.cc:1034] running on GCE node
I20240501 20:11:43.885386 29253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:43.886787 29253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:43.888015 29253 hybrid_clock.cc:648] HybridClock initialized: now 1714594303887959 us; error 51 us; skew 500 ppm
I20240501 20:11:43.888291 29253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:43.890657 29253 webserver.cc:460] Webserver started at http://127.14.106.199:35385/ using document root <none> and password file <none>
I20240501 20:11:43.891440 29253 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:43.891551 29253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:43.896544 29253 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:11:43.899150 29268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.900480 29253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:11:43.900604 29253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "212de34e3aa24fa1be12c7421df985fc"
format_stamp: "Formatted at 2024-05-01 20:11:23 on dist-test-slave-pg4x"
I20240501 20:11:43.901409 29253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:43.940464 29253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:43.941306 29253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:43.941587 29253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:43.942592 29253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:43.944109 29253 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:43.944190 29253 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.944320 29253 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:43.944377 29253 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.976418 29253 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.199:32969
I20240501 20:11:43.976433 29380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.199:32969 every 8 connection(s)
I20240501 20:11:43.977833 29253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:11:43.986968 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 29253
I20240501 20:11:43.987766 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.200:44949
--local_ip_for_outbound_sockets=127.14.106.200
--tserver_master_addrs=127.14.106.254:34283
--webserver_port=45359
--webserver_interface=127.14.106.200
--builtin_ntp_servers=127.14.106.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:11:43.994597 29381 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:43.994974 29381 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:43.995909 29381 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:43.997314 26993 ts_manager.cc:194] Re-registered known tserver with Master: 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
I20240501 20:11:43.998114 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.199:35615
I20240501 20:11:44.010022 29115 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
W20240501 20:11:44.347437 29384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:44.347697 29384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:44.347831 29384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:44.355654 29384 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:11:44.355765 29384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:44.355880 29384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.200
I20240501 20:11:44.364648 29384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.200:44949
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.106.200
--webserver_port=45359
--tserver_master_addrs=127.14.106.254:34283
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.200
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:44.365916 29384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:44.367615 29384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:44.379696 29393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:44.380187 29391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:44.380703 29390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:44.381256 29384 server_base.cc:1034] running on GCE node
I20240501 20:11:44.381793 29384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:44.382695 29384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:44.383905 29384 hybrid_clock.cc:648] HybridClock initialized: now 1714594304383862 us; error 34 us; skew 500 ppm
I20240501 20:11:44.384260 29384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:44.386425 29384 webserver.cc:460] Webserver started at http://127.14.106.200:45359/ using document root <none> and password file <none>
I20240501 20:11:44.387151 29384 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:44.387270 29384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:44.392355 29384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:11:44.394937 29399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:44.396206 29384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:11:44.396338 29384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e957d29835544ac587917cc067fb7423"
format_stamp: "Formatted at 2024-05-01 20:11:24 on dist-test-slave-pg4x"
I20240501 20:11:44.397089 29384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:44.427577 29384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:44.428368 29384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:44.428632 29384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:44.429510 29384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:44.431172 29384 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:44.431255 29384 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:44.431362 29384 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:44.431435 29384 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:44.466485 29384 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.200:44949
I20240501 20:11:44.466507 29511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.200:44949 every 8 connection(s)
I20240501 20:11:44.468156 29384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:11:44.477349 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 29384
I20240501 20:11:44.480897 29512 heartbeater.cc:347] Connected to a master server at 127.14.106.254:34283
I20240501 20:11:44.481272 29512 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:44.482206 29512 heartbeater.cc:511] Master 127.14.106.254:34283 requested a full tablet report, sending...
I20240501 20:11:44.485070 26993 ts_manager.cc:194] Re-registered known tserver with Master: e957d29835544ac587917cc067fb7423 (127.14.106.200:44949)
I20240501 20:11:44.486114 26993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.200:38275
I20240501 20:11:44.505537 29248 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
I20240501 20:11:45.002795 29381 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
I20240501 20:11:45.488977 29512 heartbeater.cc:503] Master 127.14.106.254:34283 was elected leader, sending a full tablet report...
I20240501 20:11:45.767061 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:45.800047 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:45.821359 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:45.891884 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:45.952541 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:45.966491 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:45.982565 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:46.004498 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.024266 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.028817 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:46.058246 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.065912 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.070235 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:46.116299 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:46.158058 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:46.218776 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.227927 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.236790 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:46.271103 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:46.281148 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.304759 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.330302 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:46.348569 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:46.372726 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:11:46.467871 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:46293, user_credentials={real_user=slave}} blocked reactor thread for 43888.8us
I20240501 20:11:46.547730 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.579735 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:46.606977 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:46.621613 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:46.648487 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:46.656668 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:46.657315 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.660138 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.681536 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.702477 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.719655 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.728677 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:46.740118 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:46.767177 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:46.774032 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:46.792214 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:47.568078 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:47.577908 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:47.592538 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:47.610054 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:47.659700 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:47.684396 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:47.706897 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:47.710312 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:47.743547 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:47.779338 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:47.785365 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:47.799088 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:47.810145 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:47.819506 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:47.854451 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:47.860431 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.033931 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.087240 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:48.090875 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.117974 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:48.139165 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:48.142223 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.142678 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:48.162081 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.324056 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.350330 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:48.357384 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.363756 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.369935 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:48.399571 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.400534 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.401345 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:48.409673 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:48.418857 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.418920 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:48.445252 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:48.472851 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.484333 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.493238 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:48.500481 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:49.067272 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.090682 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:49.150126 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:49.156283 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.191792 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.211578 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:49.231782 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:49.233830 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.404181 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.468145 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:49.500211 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.509357 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.531041 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:49.573210 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:49.597373 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:49.611363 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.943023 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.946566 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.953838 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:49.982926 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:49.985352 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.039373 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:50.039702 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:50.052892 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:11:50.065284 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 94605.3us
I20240501 20:11:50.075060 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:50.082835 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.109562 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.110875 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:50.171128 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:11:50.178475 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 99511.2us
I20240501 20:11:50.182369 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.204797 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.218544 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:50.325222 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.352756 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:50.415199 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.417219 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:50.475716 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.479187 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:50.498255 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:50.502868 27133 consensus_queue.cc:237] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:50.512357 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:50.524670 27265 raft_consensus.cc:1261] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:50.525296 30249 consensus_queue.cc:1035] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:50.534747 27396 raft_consensus.cc:1261] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:50.535625 30249 consensus_queue.cc:1035] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:50.579633 27330 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:43861, user_credentials={real_user=slave}} blocked reactor thread for 116450us
I20240501 20:11:50.602427 30249 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:50.602993 27265 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
W20240501 20:11:50.608377 27068 consensus_peers.cc:485] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dc5794a67374cc388da2edc47cbb113. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:50.615063 26994 catalog_manager.cc:5484] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
W20240501 20:11:50.669965 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 50298.5us
I20240501 20:11:50.672678 27396 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:50.799392 27133 consensus_queue.cc:237] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
W20240501 20:11:50.811100 27068 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78fcaa70e916442ea9c67cc762c8db6d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:50.812112 27526 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:50.813086 30249 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:50.815953 27264 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:50.818579 30249 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:50.830267 30249 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:50.831362 27526 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:50.837141 27264 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:50.838438 26994 catalog_manager.cc:5484] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:50.886327 27134 consensus_queue.cc:237] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:50.897334 27526 raft_consensus.cc:1261] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:50.898017 27396 raft_consensus.cc:1261] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:50.898131 30307 consensus_queue.cc:1035] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:50.898801 30307 consensus_queue.cc:1035] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:50.916484 27070 consensus_peers.cc:485] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 309e90bbd9f140a8beca9b3da3f75db4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:50.944178 30249 raft_consensus.cc:2932] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:50.945206 27396 raft_consensus.cc:2932] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:50.954502 26994 catalog_manager.cc:5484] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:50.974061 27526 raft_consensus.cc:2932] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:51.049814 27134 consensus_queue.cc:237] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:51.054100 27526 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:51.055897 27264 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:51.056969 30303 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:11:51.058338 27068 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78fcaa70e916442ea9c67cc762c8db6d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:51.059181 29738 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:51.066680 29738 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.071746 27264 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.079947 26994 catalog_manager.cc:5484] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.084549 27526 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.145107 30408 ts_tablet_manager.cc:902] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
W20240501 20:11:51.175140 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78fcaa70e916442ea9c67cc762c8db6d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:51.207823 30408 tablet_copy_client.cc:308] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:51.241361 27155 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 4dc5794a67374cc388da2edc47cbb113 from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:42311)
I20240501 20:11:51.241669 27155 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 4dc5794a67374cc388da2edc47cbb113 from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:42311: session id = 212de34e3aa24fa1be12c7421df985fc-4dc5794a67374cc388da2edc47cbb113
I20240501 20:11:51.242965 27396 consensus_queue.cc:237] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:51.245316 27155 tablet_copy_source_session.cc:205] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:51.250794 30408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc5794a67374cc388da2edc47cbb113. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:51.275714 27526 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:51.276603 30297 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:51.283679 27134 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:51.284273 30297 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:51.285859 27332 consensus_peers.cc:485] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5a20d47052f404986315ddb05284ad9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:51.294974 30408 tablet_copy_client.cc:783] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:51.295471 30408 tablet_copy_client.cc:647] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:51.298607 30452 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:51.299876 30408 tablet_copy_client.cc:523] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:51.304469 27526 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:51.309255 30408 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:51.322364 27133 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:51.323318 26992 catalog_manager.cc:5484] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:51.348868 30460 ts_tablet_manager.cc:902] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:51.358422 27264 consensus_queue.cc:237] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:51.365891 30408 log.cc:824] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:51.369855 27133 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:51.370512 30290 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:51.370927 27134 consensus_queue.cc:237] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:51.397667 27526 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:51.401183 27264 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:51.401801 30303 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:51.403124 27396 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:51.403935 30303 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:51.403960 30290 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:51.410567 27071 consensus_peers.cc:485] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5427fbc8a244a69974fd8d0f5f3d08d. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:51.427357 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26cb2329d176441ca9127187db2d157a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:51.427853 30460 tablet_copy_client.cc:308] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:51.429117 30303 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:51.434438 27395 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:51.437485 26992 catalog_manager.cc:5484] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:51.441169 30408 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:51.441783 30408 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:51.442345 30408 ts_tablet_manager.cc:1367] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.133s user 0.059s sys 0.025s
I20240501 20:11:51.443820 27264 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:51.448827 30408 raft_consensus.cc:348] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:51.449347 30408 raft_consensus.cc:729] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:51.450248 30408 consensus_queue.cc:260] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:51.462321 30408 ts_tablet_manager.cc:1398] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.020s user 0.012s sys 0.004s
I20240501 20:11:51.465551 27155 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 309e90bbd9f140a8beca9b3da3f75db4 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:43137)
I20240501 20:11:51.465811 27155 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 309e90bbd9f140a8beca9b3da3f75db4 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:43137: session id = e957d29835544ac587917cc067fb7423-309e90bbd9f140a8beca9b3da3f75db4
I20240501 20:11:51.469414 27155 tablet_copy_source_session.cc:205] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:51.474689 27155 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-4dc5794a67374cc388da2edc47cbb113 received from {username='slave'} at 127.14.106.199:42311
I20240501 20:11:51.474886 27155 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-4dc5794a67374cc388da2edc47cbb113 on tablet 4dc5794a67374cc388da2edc47cbb113 with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:51.480290 30408 ts_tablet_manager.cc:902] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:51.509953 30408 tablet_copy_client.cc:308] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:51.510710 27155 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 78fcaa70e916442ea9c67cc762c8db6d from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:42311)
I20240501 20:11:51.510879 27155 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 78fcaa70e916442ea9c67cc762c8db6d from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:42311: session id = 212de34e3aa24fa1be12c7421df985fc-78fcaa70e916442ea9c67cc762c8db6d
I20240501 20:11:51.513979 27155 tablet_copy_source_session.cc:205] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:51.515671 30408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78fcaa70e916442ea9c67cc762c8db6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:51.525151 30492 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.526906 30408 tablet_copy_client.cc:783] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:51.527189 30408 tablet_copy_client.cc:647] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:51.528820 30408 tablet_copy_client.cc:523] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:51.532552 30408 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:51.532718 27526 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.535830 30460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309e90bbd9f140a8beca9b3da3f75db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:51.540463 26994 catalog_manager.cc:5484] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.570052 27134 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.608165 30509 ts_tablet_manager.cc:902] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:51.610615 29336 raft_consensus.cc:1203] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:51.627476 27134 consensus_queue.cc:237] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:51.644716 27395 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:51.645510 30303 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:51.653134 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bf74c24f28546bb90e10002270e3814. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:51.654157 27264 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:51.654968 30303 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:51.658435 30408 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:51.659076 30408 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:51.659415 30408 ts_tablet_manager.cc:1367] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.127s user 0.065s sys 0.004s
I20240501 20:11:51.660677 30408 raft_consensus.cc:348] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:51.660991 30408 raft_consensus.cc:729] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:51.661278 30408 consensus_queue.cc:260] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:51.662171 30408 ts_tablet_manager.cc:1398] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:51.663811 27155 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-78fcaa70e916442ea9c67cc762c8db6d received from {username='slave'} at 127.14.106.199:42311
I20240501 20:11:51.663987 27155 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-78fcaa70e916442ea9c67cc762c8db6d on tablet 78fcaa70e916442ea9c67cc762c8db6d with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:51.686652 29738 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.687304 27396 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.695554 26992 catalog_manager.cc:5484] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.708081 27265 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:51.747341 30509 tablet_copy_client.cc:308] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:51.752878 30460 tablet_copy_client.cc:783] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:51.753428 30460 tablet_copy_client.cc:647] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:51.762459 30460 tablet_copy_client.cc:523] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:51.784037 30460 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:51.810616 27155 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 78fcaa70e916442ea9c67cc762c8db6d from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:55267)
I20240501 20:11:51.810843 27155 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 78fcaa70e916442ea9c67cc762c8db6d from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:55267: session id = d56cbcb06b9e418cb617de6a34b946b0-78fcaa70e916442ea9c67cc762c8db6d
I20240501 20:11:51.813716 27155 tablet_copy_source_session.cc:205] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:51.836077 30509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78fcaa70e916442ea9c67cc762c8db6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:51.853686 30460 log.cc:824] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:51.860277 30558 ts_tablet_manager.cc:902] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:51.868819 30564 ts_tablet_manager.cc:902] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:51.899241 30564 tablet_copy_client.cc:308] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:51.903901 30509 tablet_copy_client.cc:783] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:51.904639 30509 tablet_copy_client.cc:647] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:51.908208 30509 tablet_copy_client.cc:523] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:51.936944 30558 tablet_copy_client.cc:308] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:51.947356 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet b5a20d47052f404986315ddb05284ad9 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:33893)
I20240501 20:11:51.947692 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet b5a20d47052f404986315ddb05284ad9 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:33893: session id = 565cade0058540a4aa18d7c7a25affbe-b5a20d47052f404986315ddb05284ad9
I20240501 20:11:51.952765 27416 tablet_copy_source_session.cc:205] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:51.959142 30564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a20d47052f404986315ddb05284ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:51.964270 27133 consensus_queue.cc:237] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:51.965137 27526 consensus_queue.cc:237] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:51.968538 27265 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:51.969575 27395 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:51.971138 30307 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:11:51.972487 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bf74c24f28546bb90e10002270e3814. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:51.972875 27070 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bf74c24f28546bb90e10002270e3814. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:51.974468 30307 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:51.977344 30307 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:51.981855 27395 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:51.990476 27265 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:51.995926 30509 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:51.996313 26992 catalog_manager.cc:5484] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:52.007133 27265 raft_consensus.cc:1261] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:52.007899 29734 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:52.016949 30605 ts_tablet_manager.cc:902] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:52.026154 30564 tablet_copy_client.cc:783] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:52.027017 30564 tablet_copy_client.cc:647] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:52.033334 30564 tablet_copy_client.cc:523] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:52.045197 30564 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:52.056581 27155 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet b5427fbc8a244a69974fd8d0f5f3d08d from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:59561)
I20240501 20:11:52.056744 27155 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet b5427fbc8a244a69974fd8d0f5f3d08d from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:59561: session id = 565cade0058540a4aa18d7c7a25affbe-b5427fbc8a244a69974fd8d0f5f3d08d
I20240501 20:11:52.059880 27155 tablet_copy_source_session.cc:205] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:52.060498 27395 raft_consensus.cc:1261] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:52.061398 30558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5427fbc8a244a69974fd8d0f5f3d08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.068980 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.071259 30558 tablet_copy_client.cc:783] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:52.071558 30558 tablet_copy_client.cc:647] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:52.072582 29734 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:52.073571 30558 tablet_copy_client.cc:523] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:52.077316 30558 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:52.084321 27134 consensus_queue.cc:237] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
W20240501 20:11:52.088320 27070 consensus_peers.cc:485] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5427fbc8a244a69974fd8d0f5f3d08d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:52.088855 27265 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:52.089392 30603 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:52.103014 30605 tablet_copy_client.cc:308] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:46281
I20240501 20:11:52.105062 29336 raft_consensus.cc:1203] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:52.110069 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:52.116753 27462 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c836d9afa0f439b978bc2fe5b5e499d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:52.120532 27396 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:52.121578 30090 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:52.125270 30307 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:52.127785 27396 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
W20240501 20:11:52.131374 27071 consensus_peers.cc:485] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:52.133778 30307 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:52.141005 27264 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:52.141891 30558 log.cc:824] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:52.144407 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.145262 27394 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:52.153774 26994 catalog_manager.cc:5484] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:52.155597 26993 catalog_manager.cc:5484] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:52.175977 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:52.180750 27264 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:52.237325 30509 log.cc:824] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:52.245038 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:52.246431 27286 tablet_copy_service.cc:140] P d8b307156e1346bd9eecf655f49924c7: Received BeginTabletCopySession request for tablet 26cb2329d176441ca9127187db2d157a from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:55097)
I20240501 20:11:52.246749 27286 tablet_copy_service.cc:161] P d8b307156e1346bd9eecf655f49924c7: Beginning new tablet copy session on tablet 26cb2329d176441ca9127187db2d157a from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:55097: session id = d56cbcb06b9e418cb617de6a34b946b0-26cb2329d176441ca9127187db2d157a
I20240501 20:11:52.252278 27286 tablet_copy_source_session.cc:205] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:52.257820 30605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cb2329d176441ca9127187db2d157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.263624 30460 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:52.268543 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:52.278002 30605 tablet_copy_client.cc:783] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:52.278421 30605 tablet_copy_client.cc:647] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:52.282192 30605 tablet_copy_client.cc:523] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:52.264243 30460 tablet_bootstrap.cc:492] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:52.290485 30460 ts_tablet_manager.cc:1367] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.510s user 0.075s sys 0.012s
I20240501 20:11:52.308607 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.308590 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.327548 30605 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:52.400687 27394 consensus_queue.cc:237] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:52.403198 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.405856 30672 ts_tablet_manager.cc:902] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
W20240501 20:11:52.410125 27332 consensus_peers.cc:485] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:52.410445 27134 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:52.411114 30452 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:52.413892 27526 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:52.414544 30452 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:52.434089 30460 raft_consensus.cc:348] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:52.438582 30558 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:52.439210 30558 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:52.439687 30558 ts_tablet_manager.cc:1367] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.362s user 0.081s sys 0.047s
I20240501 20:11:52.446610 30558 raft_consensus.cc:348] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:52.449224 30558 raft_consensus.cc:729] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.450096 30558 consensus_queue.cc:260] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:52.453368 30460 raft_consensus.cc:729] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.454173 30460 consensus_queue.cc:260] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:52.465520 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:52.470544 30558 ts_tablet_manager.cc:1398] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.031s user 0.012s sys 0.000s
I20240501 20:11:52.471480 27155 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-b5427fbc8a244a69974fd8d0f5f3d08d received from {username='slave'} at 127.14.106.197:59561
I20240501 20:11:52.471632 27155 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-b5427fbc8a244a69974fd8d0f5f3d08d on tablet b5427fbc8a244a69974fd8d0f5f3d08d with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:52.485589 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.496519 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:52.511188 30700 ts_tablet_manager.cc:902] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:52.512210 30698 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.513729 27133 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.521567 26992 catalog_manager.cc:5484] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.536578 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:52.544181 30701 ts_tablet_manager.cc:902] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:52.531414 27394 consensus_queue.cc:237] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:52.560027 30699 ts_tablet_manager.cc:902] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:52.564836 30460 ts_tablet_manager.cc:1398] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.274s user 0.014s sys 0.008s
I20240501 20:11:52.568346 27264 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:52.569679 30452 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:52.580673 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:11:52.582744 27329 consensus_peers.cc:485] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5a20d47052f404986315ddb05284ad9. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:52.583076 27329 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57b4bd50bc8f4d50957bd0280224c420. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:52.589607 30672 tablet_copy_client.cc:308] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:52.593763 30701 tablet_copy_client.cc:308] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:52.594776 30699 tablet_copy_client.cc:308] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:52.595129 27526 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:52.595954 30711 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:52.608942 27546 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 6c836d9afa0f439b978bc2fe5b5e499d from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:54863)
I20240501 20:11:52.609243 27546 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 6c836d9afa0f439b978bc2fe5b5e499d from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:54863: session id = 565cade0058540a4aa18d7c7a25affbe-6c836d9afa0f439b978bc2fe5b5e499d
I20240501 20:11:52.609990 30711 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.615489 27546 tablet_copy_source_session.cc:205] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:52.615094 27524 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.624513 26993 catalog_manager.cc:5484] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.625547 30672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c836d9afa0f439b978bc2fe5b5e499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.641042 27264 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.658862 30672 tablet_copy_client.cc:783] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:52.659353 30672 tablet_copy_client.cc:647] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:52.662464 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.689882 30672 tablet_copy_client.cc:523] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:52.690493 27526 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.694839 30672 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:52.698189 27155 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 8bf74c24f28546bb90e10002270e3814 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:55267)
I20240501 20:11:52.698444 27155 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 8bf74c24f28546bb90e10002270e3814 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:55267: session id = d56cbcb06b9e418cb617de6a34b946b0-8bf74c24f28546bb90e10002270e3814
I20240501 20:11:52.701529 30700 tablet_copy_client.cc:308] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:52.704455 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session e957d29835544ac587917cc067fb7423-309e90bbd9f140a8beca9b3da3f75db4 received from {username='slave'} at 127.14.106.200:43137
I20240501 20:11:52.704622 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session e957d29835544ac587917cc067fb7423-309e90bbd9f140a8beca9b3da3f75db4 on tablet 309e90bbd9f140a8beca9b3da3f75db4 with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:52.711201 27153 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet b5427fbc8a244a69974fd8d0f5f3d08d from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:43137)
I20240501 20:11:52.711378 27153 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet b5427fbc8a244a69974fd8d0f5f3d08d from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:43137: session id = e957d29835544ac587917cc067fb7423-b5427fbc8a244a69974fd8d0f5f3d08d
I20240501 20:11:52.712167 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 8bf74c24f28546bb90e10002270e3814 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:43137)
I20240501 20:11:52.712342 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 8bf74c24f28546bb90e10002270e3814 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:43137: session id = e957d29835544ac587917cc067fb7423-8bf74c24f28546bb90e10002270e3814
I20240501 20:11:52.701887 27155 tablet_copy_source_session.cc:205] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:52.714592 27153 tablet_copy_source_session.cc:205] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:52.723464 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5427fbc8a244a69974fd8d0f5f3d08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.725848 27526 consensus_queue.cc:237] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:52.727025 27154 tablet_copy_source_session.cc:205] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:52.728221 30701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf74c24f28546bb90e10002270e3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.728994 30564 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:52.737594 30564 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:52.737915 30564 ts_tablet_manager.cc:1367] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.693s user 0.052s sys 0.013s
I20240501 20:11:52.739073 30564 raft_consensus.cc:348] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:52.739408 30564 raft_consensus.cc:729] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.739674 30701 tablet_copy_client.cc:783] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:52.739728 30564 consensus_queue.cc:260] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:52.740280 30701 tablet_copy_client.cc:647] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:52.744926 30701 tablet_copy_client.cc:523] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:52.746424 30700 tablet_copy_client.cc:783] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:52.746815 30700 tablet_copy_client.cc:647] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:52.747857 30564 ts_tablet_manager.cc:1398] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:11:52.750190 30700 tablet_copy_client.cc:523] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:52.755016 30701 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:52.757380 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-b5a20d47052f404986315ddb05284ad9 received from {username='slave'} at 127.14.106.197:33893
I20240501 20:11:52.762719 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:52.764127 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-b5a20d47052f404986315ddb05284ad9 on tablet b5a20d47052f404986315ddb05284ad9 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:52.767386 30700 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:52.769726 30509 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:52.770306 30509 tablet_bootstrap.cc:492] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:52.769636 29069 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:52.770843 30509 ts_tablet_manager.cc:1367] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.775s user 0.058s sys 0.016s
I20240501 20:11:52.774377 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf74c24f28546bb90e10002270e3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.790912 30699 tablet_copy_client.cc:783] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:52.791220 30699 tablet_copy_client.cc:647] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:52.796756 27134 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:52.799085 30090 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:52.805657 30605 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:52.807072 27394 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:52.807268 30605 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:52.807787 30630 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:52.808571 30605 ts_tablet_manager.cc:1367] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.503s user 0.048s sys 0.006s
I20240501 20:11:52.817745 30699 tablet_copy_client.cc:523] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:52.821858 30699 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:52.828496 30605 raft_consensus.cc:348] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:52.831063 30605 raft_consensus.cc:729] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.831878 30605 consensus_queue.cc:260] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:52.833823 29466 raft_consensus.cc:1203] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:52.841231 30605 ts_tablet_manager.cc:1398] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20240501 20:11:52.830865 30509 raft_consensus.cc:348] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:52.841675 30509 raft_consensus.cc:729] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.842098 30509 consensus_queue.cc:260] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:52.843819 30509 ts_tablet_manager.cc:1398] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.073s user 0.007s sys 0.001s
I20240501 20:11:52.850076 27286 tablet_copy_service.cc:342] P d8b307156e1346bd9eecf655f49924c7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-26cb2329d176441ca9127187db2d157a received from {username='slave'} at 127.14.106.198:55097
I20240501 20:11:52.850283 27286 tablet_copy_service.cc:433] P d8b307156e1346bd9eecf655f49924c7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-26cb2329d176441ca9127187db2d157a on tablet 26cb2329d176441ca9127187db2d157a with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:52.851680 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-78fcaa70e916442ea9c67cc762c8db6d received from {username='slave'} at 127.14.106.198:55267
I20240501 20:11:52.851835 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-78fcaa70e916442ea9c67cc762c8db6d on tablet 78fcaa70e916442ea9c67cc762c8db6d with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:52.870563 27133 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.878782 30630 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.886157 27394 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.893205 26993 catalog_manager.cc:5484] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.944943 30764 ts_tablet_manager.cc:902] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
W20240501 20:11:52.958379 27459 consensus_peers.cc:485] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f034a679870b44caa238f04a8b49d302. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:52.965889 30700 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:52.975525 30701 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:52.976203 30701 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:52.976485 30701 ts_tablet_manager.cc:1367] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.222s user 0.058s sys 0.002s
I20240501 20:11:52.978070 30701 raft_consensus.cc:348] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:52.978446 30701 raft_consensus.cc:729] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.979197 30701 consensus_queue.cc:260] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:52.978848 30700 tablet_bootstrap.cc:492] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:52.979817 30700 ts_tablet_manager.cc:1367] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.213s user 0.047s sys 0.013s
I20240501 20:11:52.980729 30701 ts_tablet_manager.cc:1398] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:11:52.989377 29069 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:52.990339 29466 raft_consensus.cc:1203] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:53.001602 30700 raft_consensus.cc:348] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:53.001981 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session e957d29835544ac587917cc067fb7423-8bf74c24f28546bb90e10002270e3814 received from {username='slave'} at 127.14.106.200:43137
I20240501 20:11:53.002142 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session e957d29835544ac587917cc067fb7423-8bf74c24f28546bb90e10002270e3814 on tablet 8bf74c24f28546bb90e10002270e3814 with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:53.005764 30700 raft_consensus.cc:729] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.005934 30672 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.008714 30672 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:53.009017 30672 ts_tablet_manager.cc:1367] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.314s user 0.044s sys 0.010s
I20240501 20:11:53.009083 30700 consensus_queue.cc:260] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:53.011340 30700 ts_tablet_manager.cc:1398] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.031s user 0.000s sys 0.001s
I20240501 20:11:53.011548 30672 raft_consensus.cc:348] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:53.011809 30672 raft_consensus.cc:729] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.011965 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session e957d29835544ac587917cc067fb7423-b5427fbc8a244a69974fd8d0f5f3d08d received from {username='slave'} at 127.14.106.200:43137
I20240501 20:11:53.012122 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session e957d29835544ac587917cc067fb7423-b5427fbc8a244a69974fd8d0f5f3d08d on tablet b5427fbc8a244a69974fd8d0f5f3d08d with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:53.012552 30672 consensus_queue.cc:260] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:53.014815 30764 tablet_copy_client.cc:308] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:53.015319 30672 ts_tablet_manager.cc:1398] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:11:53.017159 27546 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-6c836d9afa0f439b978bc2fe5b5e499d received from {username='slave'} at 127.14.106.197:54863
I20240501 20:11:53.017359 27546 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-6c836d9afa0f439b978bc2fe5b5e499d on tablet 6c836d9afa0f439b978bc2fe5b5e499d with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:53.053839 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet b5a20d47052f404986315ddb05284ad9 from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:58117)
I20240501 20:11:53.054033 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet b5a20d47052f404986315ddb05284ad9 from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:58117: session id = 212de34e3aa24fa1be12c7421df985fc-b5a20d47052f404986315ddb05284ad9
I20240501 20:11:53.057520 27416 tablet_copy_source_session.cc:205] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:53.059794 30764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a20d47052f404986315ddb05284ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.066145 27525 consensus_queue.cc:237] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:53.067070 30764 tablet_copy_client.cc:783] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:53.067363 30764 tablet_copy_client.cc:647] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:53.069145 30764 tablet_copy_client.cc:523] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:53.072536 30764 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:53.075798 29203 raft_consensus.cc:1203] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:53.085855 29203 raft_consensus.cc:1203] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 20:11:53.097965 27462 consensus_peers.cc:485] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f02a619bc1584d53ac902c8d8baff976. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:53.099017 27134 raft_consensus.cc:1261] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.099530 30699 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.099987 30699 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:53.100286 30699 ts_tablet_manager.cc:1367] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.279s user 0.057s sys 0.006s
I20240501 20:11:53.100454 30746 consensus_queue.cc:1035] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:53.103679 27394 raft_consensus.cc:1261] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.105054 30745 consensus_queue.cc:1035] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:53.131081 30699 raft_consensus.cc:348] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:53.131486 30699 raft_consensus.cc:729] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.131899 30699 consensus_queue.cc:260] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:53.132704 30699 ts_tablet_manager.cc:1398] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20240501 20:11:53.133471 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-8bf74c24f28546bb90e10002270e3814 received from {username='slave'} at 127.14.106.198:55267
I20240501 20:11:53.133657 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-8bf74c24f28546bb90e10002270e3814 on tablet 8bf74c24f28546bb90e10002270e3814 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:53.143599 30745 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:53.147532 27133 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:53.157616 26993 catalog_manager.cc:5484] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:53.158528 27394 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:53.197681 30800 ts_tablet_manager.cc:902] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:53.221809 30800 tablet_copy_client.cc:308] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:53.222725 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet 57b4bd50bc8f4d50957bd0280224c420 from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:58117)
I20240501 20:11:53.222888 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet 57b4bd50bc8f4d50957bd0280224c420 from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:58117: session id = 212de34e3aa24fa1be12c7421df985fc-57b4bd50bc8f4d50957bd0280224c420
I20240501 20:11:53.225831 27416 tablet_copy_source_session.cc:205] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:53.227788 30800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b4bd50bc8f4d50957bd0280224c420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.248829 30800 tablet_copy_client.cc:783] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:53.249168 30800 tablet_copy_client.cc:647] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:53.251266 30800 tablet_copy_client.cc:523] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:53.271106 30800 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:53.281358 27394 consensus_queue.cc:237] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
W20240501 20:11:53.285115 27329 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:53.287217 27525 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:53.293303 30827 ts_tablet_manager.cc:902] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:53.296106 30764 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.296617 30764 tablet_bootstrap.cc:492] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:53.296936 30764 ts_tablet_manager.cc:1367] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.224s user 0.063s sys 0.020s
I20240501 20:11:53.297924 30764 raft_consensus.cc:348] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:53.298285 30764 raft_consensus.cc:729] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.304338 30827 tablet_copy_client.cc:308] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:53.305233 30764 consensus_queue.cc:260] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:53.314734 27546 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet f034a679870b44caa238f04a8b49d302 from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:37233)
I20240501 20:11:53.314961 27546 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet f034a679870b44caa238f04a8b49d302 from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:37233: session id = 212de34e3aa24fa1be12c7421df985fc-f034a679870b44caa238f04a8b49d302
I20240501 20:11:53.319411 27546 tablet_copy_source_session.cc:205] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:53.321182 30827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f034a679870b44caa238f04a8b49d302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.331508 30827 tablet_copy_client.cc:783] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:53.334708 30827 tablet_copy_client.cc:647] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:53.337604 30827 tablet_copy_client.cc:523] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:53.339367 30711 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:53.340911 27264 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:53.341535 30711 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:53.349126 30827 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:53.350116 30711 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.354674 30764 ts_tablet_manager.cc:1398] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.058s user 0.005s sys 0.014s
I20240501 20:11:53.356468 27264 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.358762 26993 catalog_manager.cc:5484] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.361763 27525 consensus_queue.cc:237] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:53.380218 27526 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.380702 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-b5a20d47052f404986315ddb05284ad9 received from {username='slave'} at 127.14.106.199:58117
I20240501 20:11:53.380853 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-b5a20d47052f404986315ddb05284ad9 on tablet b5a20d47052f404986315ddb05284ad9 with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:53.392889 27134 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.395661 27264 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.396298 30745 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:53.401609 30745 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:53.403625 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57b4bd50bc8f4d50957bd0280224c420. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:53.413268 29336 raft_consensus.cc:1203] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:53.436642 30746 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:53.438848 29070 raft_consensus.cc:1203] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:53.442863 27133 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:53.443244 26993 catalog_manager.cc:5484] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:53.437512 27266 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
W20240501 20:11:53.447165 27462 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74c414c2eed045bda9d85fe5c2fea654. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:53.478830 29466 raft_consensus.cc:1203] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:53.513388 30800 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.513919 30800 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:53.514348 30800 ts_tablet_manager.cc:1367] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.247s user 0.053s sys 0.008s
I20240501 20:11:53.527743 30800 raft_consensus.cc:348] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:53.528096 30800 raft_consensus.cc:729] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.530586 27525 consensus_queue.cc:237] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:53.536983 30800 consensus_queue.cc:260] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:53.544239 30800 ts_tablet_manager.cc:1398] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.030s user 0.001s sys 0.004s
I20240501 20:11:53.545336 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-57b4bd50bc8f4d50957bd0280224c420 received from {username='slave'} at 127.14.106.199:58117
I20240501 20:11:53.545518 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-57b4bd50bc8f4d50957bd0280224c420 on tablet 57b4bd50bc8f4d50957bd0280224c420 with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:53.554092 27266 raft_consensus.cc:1261] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.555358 27394 raft_consensus.cc:1261] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.556660 29203 raft_consensus.cc:1203] T 8bf74c24f28546bb90e10002270e3814 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:53.558858 30746 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:53.561566 30746 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:53.571641 30827 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.572252 30827 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:53.572593 30827 ts_tablet_manager.cc:1367] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.224s user 0.052s sys 0.006s
I20240501 20:11:53.579449 30827 raft_consensus.cc:348] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:53.579737 30827 raft_consensus.cc:729] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.583231 30827 consensus_queue.cc:260] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:53.585835 30827 ts_tablet_manager.cc:1398] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:11:53.589726 27546 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-f034a679870b44caa238f04a8b49d302 received from {username='slave'} at 127.14.106.199:37233
I20240501 20:11:53.589874 27546 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-f034a679870b44caa238f04a8b49d302 on tablet f034a679870b44caa238f04a8b49d302 with peer 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.591706 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bb13c1dce8640b28063daa0132d2896. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:53.602109 30897 ts_tablet_manager.cc:902] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:53.610401 30746 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.613075 27265 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.620229 26993 catalog_manager.cc:5484] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.628341 27394 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.666863 30897 tablet_copy_client.cc:308] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:53.669138 27546 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet f02a619bc1584d53ac902c8d8baff976 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:54863)
I20240501 20:11:53.669370 27546 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet f02a619bc1584d53ac902c8d8baff976 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:54863: session id = 565cade0058540a4aa18d7c7a25affbe-f02a619bc1584d53ac902c8d8baff976
I20240501 20:11:53.673090 27546 tablet_copy_source_session.cc:205] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:53.675297 30897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02a619bc1584d53ac902c8d8baff976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.703181 30897 tablet_copy_client.cc:783] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:53.710098 30897 tablet_copy_client.cc:647] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:53.712626 30897 tablet_copy_client.cc:523] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:53.721050 30897 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:53.748014 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:53.749156 27525 consensus_queue.cc:237] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:53.752061 29336 raft_consensus.cc:1203] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:53.772246 27266 raft_consensus.cc:1261] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.773558 30745 consensus_queue.cc:1035] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:53.783320 27133 raft_consensus.cc:1261] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.784042 30746 consensus_queue.cc:1035] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:53.803012 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:53.810957 30746 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:53.816869 27134 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:53.823082 26992 catalog_manager.cc:5484] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:53.830082 27265 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:53.839720 27525 consensus_queue.cc:237] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:53.848877 29336 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:53.894055 30969 ts_tablet_manager.cc:902] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:53.937274 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:53.960088 30958 ts_tablet_manager.cc:902] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:53.964202 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:53.977880 30897 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.979147 30897 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:53.979476 30897 ts_tablet_manager.cc:1367] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.259s user 0.062s sys 0.005s
I20240501 20:11:53.980232 30897 raft_consensus.cc:348] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:53.980517 30897 raft_consensus.cc:729] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.983109 30897 consensus_queue.cc:260] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:53.989161 27134 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.989684 30745 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:53.995837 30897 ts_tablet_manager.cc:1398] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.016s user 0.001s sys 0.004s
I20240501 20:11:53.996537 27546 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-f02a619bc1584d53ac902c8d8baff976 received from {username='slave'} at 127.14.106.197:54863
I20240501 20:11:53.996716 27546 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-f02a619bc1584d53ac902c8d8baff976 on tablet f02a619bc1584d53ac902c8d8baff976 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:53.997256 27266 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:53.998154 30745 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.010655 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:11:54.014039 27461 consensus_peers.cc:485] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0e88937e1ec49bfbaffe535e445e40a. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:54.014253 27461 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9721fc9afa4741b78823e293514e6265. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.015444 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:54.017704 30745 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:54.023957 27133 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:54.030758 27266 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:54.031802 26992 catalog_manager.cc:5484] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:54.037142 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:54.043478 30969 tablet_copy_client.cc:308] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:54.128473 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:54.135421 27525 consensus_queue.cc:237] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:54.155869 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet 57b4bd50bc8f4d50957bd0280224c420 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:51121)
I20240501 20:11:54.156073 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet 57b4bd50bc8f4d50957bd0280224c420 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:51121: session id = d56cbcb06b9e418cb617de6a34b946b0-57b4bd50bc8f4d50957bd0280224c420
I20240501 20:11:54.160105 27416 tablet_copy_source_session.cc:205] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:54.161729 30969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b4bd50bc8f4d50957bd0280224c420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.171741 30958 tablet_copy_client.cc:308] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:54.173043 27546 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 74c414c2eed045bda9d85fe5c2fea654 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:54863)
I20240501 20:11:54.173230 27546 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 74c414c2eed045bda9d85fe5c2fea654 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:54863: session id = 565cade0058540a4aa18d7c7a25affbe-74c414c2eed045bda9d85fe5c2fea654
I20240501 20:11:54.176311 27546 tablet_copy_source_session.cc:205] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:54.178489 30969 tablet_copy_client.cc:783] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:54.178843 30969 tablet_copy_client.cc:647] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:54.180907 30969 tablet_copy_client.cc:523] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:54.181641 27394 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.181766 31004 ts_tablet_manager.cc:902] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
W20240501 20:11:54.182174 27461 consensus_peers.cc:485] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af693dc19f844b12973b000093021619. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.182703 30745 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.189110 27266 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.189898 30996 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.190431 30969 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:54.191943 30958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c414c2eed045bda9d85fe5c2fea654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.216243 30996 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:54.219403 27265 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:54.236269 30958 tablet_copy_client.cc:783] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:54.236570 30958 tablet_copy_client.cc:647] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:54.239081 30958 tablet_copy_client.cc:523] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:54.241715 27394 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:54.247362 26992 catalog_manager.cc:5484] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:54.279065 30958 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:54.289533 31024 ts_tablet_manager.cc:902] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:54.298534 31004 tablet_copy_client.cc:308] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:54.310083 27546 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 7bb13c1dce8640b28063daa0132d2896 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:51531)
I20240501 20:11:54.310261 27546 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 7bb13c1dce8640b28063daa0132d2896 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:51531: session id = d56cbcb06b9e418cb617de6a34b946b0-7bb13c1dce8640b28063daa0132d2896
I20240501 20:11:54.313256 27546 tablet_copy_source_session.cc:205] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:54.328480 31024 tablet_copy_client.cc:308] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:54.352223 27546 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet c0e88937e1ec49bfbaffe535e445e40a from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:55441)
I20240501 20:11:54.352437 27546 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet c0e88937e1ec49bfbaffe535e445e40a from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:55441: session id = e957d29835544ac587917cc067fb7423-c0e88937e1ec49bfbaffe535e445e40a
I20240501 20:11:54.356375 27546 tablet_copy_source_session.cc:205] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:54.357833 31024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e88937e1ec49bfbaffe535e445e40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.364463 31024 tablet_copy_client.cc:783] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:54.364779 31024 tablet_copy_client.cc:647] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:54.370110 31024 tablet_copy_client.cc:523] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:54.374450 31024 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:54.377152 31004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb13c1dce8640b28063daa0132d2896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.403661 29070 raft_consensus.cc:1203] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:54.442590 27133 consensus_queue.cc:237] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
W20240501 20:11:54.446094 27071 consensus_peers.cc:485] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f8902b7b0b649edb10f362713ad36fe. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.446725 27266 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.449374 30307 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.453320 27394 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.454784 30307 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.472673 30969 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:54.473213 30969 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:54.473156 27395 consensus_queue.cc:237] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:54.473528 30969 ts_tablet_manager.cc:1367] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.283s user 0.079s sys 0.035s
I20240501 20:11:54.474328 30969 raft_consensus.cc:348] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:54.474588 30969 raft_consensus.cc:729] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.474869 30969 consensus_queue.cc:260] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:54.482873 31024 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:54.483372 31024 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:54.483665 31024 ts_tablet_manager.cc:1367] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.109s user 0.049s sys 0.006s
I20240501 20:11:54.484445 31024 raft_consensus.cc:348] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:54.484673 31024 raft_consensus.cc:729] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.485497 31024 consensus_queue.cc:260] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:54.497786 30969 ts_tablet_manager.cc:1398] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.024s user 0.007s sys 0.004s
I20240501 20:11:54.497846 30307 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
W20240501 20:11:54.502424 27329 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c619a2235c9248eeb6b2e1d78583d1ba. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.504417 27524 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.505038 30831 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.505878 27265 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.507354 30831 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.511173 27264 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:54.515489 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-57b4bd50bc8f4d50957bd0280224c420 received from {username='slave'} at 127.14.106.198:51121
I20240501 20:11:54.515619 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-57b4bd50bc8f4d50957bd0280224c420 on tablet 57b4bd50bc8f4d50957bd0280224c420 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:54.518112 26994 catalog_manager.cc:5484] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:54.527148 27395 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:54.539136 30831 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:54.539610 27265 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:54.547302 26994 catalog_manager.cc:5484] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:54.548472 31004 tablet_copy_client.cc:783] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:54.549031 31024 ts_tablet_manager.cc:1398] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.065s user 0.018s sys 0.032s
I20240501 20:11:54.550155 31004 tablet_copy_client.cc:647] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:54.557284 27546 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session e957d29835544ac587917cc067fb7423-c0e88937e1ec49bfbaffe535e445e40a received from {username='slave'} at 127.14.106.200:55441
I20240501 20:11:54.557466 27546 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session e957d29835544ac587917cc067fb7423-c0e88937e1ec49bfbaffe535e445e40a on tablet c0e88937e1ec49bfbaffe535e445e40a with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:54.557475 27524 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:54.564680 31004 tablet_copy_client.cc:523] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:54.569272 31024 ts_tablet_manager.cc:902] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:54.573288 30958 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:54.573716 30958 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:54.574002 30958 ts_tablet_manager.cc:1367] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.295s user 0.054s sys 0.000s
I20240501 20:11:54.574791 30958 raft_consensus.cc:348] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:54.575054 30958 raft_consensus.cc:729] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.583714 30958 consensus_queue.cc:260] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:54.586910 31024 tablet_copy_client.cc:308] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:54.588116 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 9721fc9afa4741b78823e293514e6265 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:55441)
I20240501 20:11:54.588310 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 9721fc9afa4741b78823e293514e6265 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:55441: session id = e957d29835544ac587917cc067fb7423-9721fc9afa4741b78823e293514e6265
I20240501 20:11:54.592216 27545 tablet_copy_source_session.cc:205] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:54.594496 31024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9721fc9afa4741b78823e293514e6265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.600353 30958 ts_tablet_manager.cc:1398] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20240501 20:11:54.601245 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-74c414c2eed045bda9d85fe5c2fea654 received from {username='slave'} at 127.14.106.197:54863
I20240501 20:11:54.601410 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-74c414c2eed045bda9d85fe5c2fea654 on tablet 74c414c2eed045bda9d85fe5c2fea654 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:54.606801 31004 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:54.615684 31024 tablet_copy_client.cc:783] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:54.616286 31024 tablet_copy_client.cc:647] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:54.630568 31024 tablet_copy_client.cc:523] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:54.641304 31024 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:54.646981 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:54.696909 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:54.728873 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:54.799439 31129 ts_tablet_manager.cc:902] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:54.809916 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:54.839452 27524 consensus_queue.cc:237] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:54.845566 29070 raft_consensus.cc:1203] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:54.856931 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:54.857513 31129 tablet_copy_client.cc:308] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:54.866034 27133 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:54.866117 27459 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7f03e02d13f44d188e5eb2774771a0d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.867605 30745 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.869611 31004 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:54.870158 31004 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:54.870452 31004 ts_tablet_manager.cc:1367] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.264s user 0.047s sys 0.011s
I20240501 20:11:54.871445 31004 raft_consensus.cc:348] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:54.871789 31004 raft_consensus.cc:729] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.872191 31004 consensus_queue.cc:260] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:54.873000 31004 ts_tablet_manager.cc:1398] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:54.873315 27395 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.873910 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-7bb13c1dce8640b28063daa0132d2896 received from {username='slave'} at 127.14.106.198:51531
I20240501 20:11:54.874068 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-7bb13c1dce8640b28063daa0132d2896 on tablet 7bb13c1dce8640b28063daa0132d2896 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:54.874040 30745 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.875557 29203 raft_consensus.cc:1203] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:54.875597 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet af693dc19f844b12973b000093021619 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:55441)
I20240501 20:11:54.875787 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet af693dc19f844b12973b000093021619 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:55441: session id = e957d29835544ac587917cc067fb7423-af693dc19f844b12973b000093021619
I20240501 20:11:54.879832 27545 tablet_copy_source_session.cc:205] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:54.891655 31129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693dc19f844b12973b000093021619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.899667 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:54.902546 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:54.903319 29466 raft_consensus.cc:1203] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:54.905151 31129 tablet_copy_client.cc:783] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:54.905489 31129 tablet_copy_client.cc:647] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:54.916945 27524 consensus_queue.cc:237] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:54.921003 27394 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:54.922123 30866 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:54.928155 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:11:54.929252 27461 consensus_peers.cc:485] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.929736 31129 tablet_copy_client.cc:523] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:54.930054 27266 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:54.930802 30866 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:11:54.931895 27462 consensus_peers.cc:485] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af693dc19f844b12973b000093021619. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.933574 30866 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:54.937296 31129 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:54.940354 30958 ts_tablet_manager.cc:902] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:54.945377 27266 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:54.952108 31024 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:54.955076 31024 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:54.955463 31024 ts_tablet_manager.cc:1367] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.314s user 0.071s sys 0.005s
I20240501 20:11:54.956357 31024 raft_consensus.cc:348] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:54.962707 31024 raft_consensus.cc:729] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.966210 27396 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:54.967981 31024 consensus_queue.cc:260] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:54.971585 31024 ts_tablet_manager.cc:1398] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 20:11:54.973625 26992 catalog_manager.cc:5484] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:54.978983 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session e957d29835544ac587917cc067fb7423-9721fc9afa4741b78823e293514e6265 received from {username='slave'} at 127.14.106.200:55441
I20240501 20:11:54.979156 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session e957d29835544ac587917cc067fb7423-9721fc9afa4741b78823e293514e6265 on tablet 9721fc9afa4741b78823e293514e6265 with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:54.983098 29466 raft_consensus.cc:1203] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:55.041841 30745 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:55.047267 27134 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:55.055120 26994 catalog_manager.cc:5484] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:55.055596 27396 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:55.063710 30958 tablet_copy_client.cc:308] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:55.068297 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 3f8902b7b0b649edb10f362713ad36fe from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:59561)
I20240501 20:11:55.068543 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 3f8902b7b0b649edb10f362713ad36fe from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:59561: session id = 565cade0058540a4aa18d7c7a25affbe-3f8902b7b0b649edb10f362713ad36fe
I20240501 20:11:55.073060 27154 tablet_copy_source_session.cc:205] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:55.074417 30958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8902b7b0b649edb10f362713ad36fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:55.089090 31174 ts_tablet_manager.cc:902] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:55.106340 30958 tablet_copy_client.cc:783] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:55.106600 30958 tablet_copy_client.cc:647] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:55.111431 30958 tablet_copy_client.cc:523] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:55.122434 29203 raft_consensus.cc:1203] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:55.124711 30958 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:55.125180 27524 consensus_queue.cc:237] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:55.139009 31174 tablet_copy_client.cc:308] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:55.140079 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet c619a2235c9248eeb6b2e1d78583d1ba from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:58117)
I20240501 20:11:55.140233 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet c619a2235c9248eeb6b2e1d78583d1ba from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:58117: session id = 212de34e3aa24fa1be12c7421df985fc-c619a2235c9248eeb6b2e1d78583d1ba
I20240501 20:11:55.143757 27416 tablet_copy_source_session.cc:205] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:55.145546 31174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c619a2235c9248eeb6b2e1d78583d1ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:11:55.153196 27459 consensus_peers.cc:485] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00e3777032eb4d9c80f47855e5f9f87a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:55.155141 27395 raft_consensus.cc:1261] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:55.155851 31171 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:55.158934 27133 raft_consensus.cc:1261] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:55.160303 31174 tablet_copy_client.cc:783] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:55.160344 31171 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:55.160670 31174 tablet_copy_client.cc:647] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:55.162357 31174 tablet_copy_client.cc:523] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:55.166690 31174 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:55.190703 31171 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:55.196673 27396 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:55.205950 27133 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:55.209800 26994 catalog_manager.cc:5484] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:55.222874 31129 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:55.223310 31129 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:55.223610 31129 ts_tablet_manager.cc:1367] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.286s user 0.057s sys 0.000s
I20240501 20:11:55.224404 31129 raft_consensus.cc:348] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:55.224659 31129 raft_consensus.cc:729] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:11:55.229112 31129 consensus_queue.cc:260] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:55.231552 31129 ts_tablet_manager.cc:1398] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:11:55.232358 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session e957d29835544ac587917cc067fb7423-af693dc19f844b12973b000093021619 received from {username='slave'} at 127.14.106.200:55441
I20240501 20:11:55.232506 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session e957d29835544ac587917cc067fb7423-af693dc19f844b12973b000093021619 on tablet af693dc19f844b12973b000093021619 with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:55.296393 30958 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:55.296866 30958 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:55.297160 30958 ts_tablet_manager.cc:1367] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.173s user 0.061s sys 0.001s
I20240501 20:11:55.298175 30958 raft_consensus.cc:348] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:55.298483 30958 raft_consensus.cc:729] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:55.298810 30958 consensus_queue.cc:260] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:55.300391 30958 ts_tablet_manager.cc:1398] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.003s
I20240501 20:11:55.302670 27524 consensus_queue.cc:237] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:55.306723 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-3f8902b7b0b649edb10f362713ad36fe received from {username='slave'} at 127.14.106.197:59561
I20240501 20:11:55.306912 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-3f8902b7b0b649edb10f362713ad36fe on tablet 3f8902b7b0b649edb10f362713ad36fe with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:55.317585 29466 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:11:55.318184 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9721fc9afa4741b78823e293514e6265. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:55.318378 27133 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:55.319945 30745 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:55.320657 31211 ts_tablet_manager.cc:902] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:55.321069 31171 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:55.325526 27266 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:55.335718 31171 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:55.336432 31211 tablet_copy_client.cc:308] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:55.337244 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet e7f03e02d13f44d188e5eb2774771a0d from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:37233)
I20240501 20:11:55.337419 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet e7f03e02d13f44d188e5eb2774771a0d from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:37233: session id = 212de34e3aa24fa1be12c7421df985fc-e7f03e02d13f44d188e5eb2774771a0d
I20240501 20:11:55.340966 27545 tablet_copy_source_session.cc:205] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:55.342595 31211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f03e02d13f44d188e5eb2774771a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:55.365172 31211 tablet_copy_client.cc:783] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:55.365463 31211 tablet_copy_client.cc:647] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:55.366153 31171 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.370198 31211 tablet_copy_client.cc:523] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:55.371901 29467 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.375451 27266 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.379513 31211 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:55.388473 26994 catalog_manager.cc:5484] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.402539 27133 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.440995 29466 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:55.449074 30958 ts_tablet_manager.cc:902] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:55.450325 30958 tablet_copy_client.cc:308] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:55.451162 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet af693dc19f844b12973b000093021619 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:54863)
I20240501 20:11:55.451370 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet af693dc19f844b12973b000093021619 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:54863: session id = 565cade0058540a4aa18d7c7a25affbe-af693dc19f844b12973b000093021619
I20240501 20:11:55.454836 27545 tablet_copy_source_session.cc:205] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:55.456739 30958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693dc19f844b12973b000093021619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:55.463078 31174 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:55.463687 31174 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:55.464040 31174 ts_tablet_manager.cc:1367] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.297s user 0.070s sys 0.026s
I20240501 20:11:55.470784 30958 tablet_copy_client.cc:783] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:55.471073 30958 tablet_copy_client.cc:647] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:55.473011 30958 tablet_copy_client.cc:523] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:55.476929 30958 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:55.484740 31174 raft_consensus.cc:348] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:55.485009 31174 raft_consensus.cc:729] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:55.489156 31174 consensus_queue.cc:260] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:55.490180 29070 raft_consensus.cc:1203] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:55.498169 31174 ts_tablet_manager.cc:1398] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20240501 20:11:55.501932 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-c619a2235c9248eeb6b2e1d78583d1ba received from {username='slave'} at 127.14.106.199:58117
I20240501 20:11:55.502090 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-c619a2235c9248eeb6b2e1d78583d1ba on tablet c619a2235c9248eeb6b2e1d78583d1ba with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:55.576231 27133 consensus_queue.cc:237] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:55.586304 29336 raft_consensus.cc:1203] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 20:11:55.587147 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5272c34458af43bebb5ba5baefac7137. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:55.588178 27266 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:55.588578 27524 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:55.589222 31064 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:55.589772 31064 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:55.622149 31211 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:55.622807 31211 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:55.623196 31211 ts_tablet_manager.cc:1367] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.244s user 0.045s sys 0.012s
I20240501 20:11:55.624215 31211 raft_consensus.cc:348] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:55.624605 31211 raft_consensus.cc:729] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:55.625041 31211 consensus_queue.cc:260] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:55.625978 31211 ts_tablet_manager.cc:1398] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:11:55.627440 30307 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.628114 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-e7f03e02d13f44d188e5eb2774771a0d received from {username='slave'} at 127.14.106.199:37233
I20240501 20:11:55.628242 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-e7f03e02d13f44d188e5eb2774771a0d on tablet e7f03e02d13f44d188e5eb2774771a0d with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:55.634464 27524 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.644583 27266 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.647599 26992 catalog_manager.cc:5484] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:55.673866 31174 ts_tablet_manager.cc:902] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:55.695262 31174 tablet_copy_client.cc:308] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:55.696219 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 00e3777032eb4d9c80f47855e5f9f87a from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:37233)
I20240501 20:11:55.696419 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 00e3777032eb4d9c80f47855e5f9f87a from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:37233: session id = 212de34e3aa24fa1be12c7421df985fc-00e3777032eb4d9c80f47855e5f9f87a
I20240501 20:11:55.699751 27545 tablet_copy_source_session.cc:205] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:55.702090 31174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e3777032eb4d9c80f47855e5f9f87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:55.710955 30958 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:55.711409 30958 tablet_bootstrap.cc:492] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:55.711717 30958 ts_tablet_manager.cc:1367] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.235s user 0.065s sys 0.005s
I20240501 20:11:55.712589 30958 raft_consensus.cc:348] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:55.712852 30958 raft_consensus.cc:729] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:55.713127 30958 consensus_queue.cc:260] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:55.718495 30958 ts_tablet_manager.cc:1398] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:11:55.720062 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-af693dc19f844b12973b000093021619 received from {username='slave'} at 127.14.106.197:54863
I20240501 20:11:55.720220 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-af693dc19f844b12973b000093021619 on tablet af693dc19f844b12973b000093021619 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:55.737646 31174 tablet_copy_client.cc:783] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:55.738159 31174 tablet_copy_client.cc:647] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:55.740191 31174 tablet_copy_client.cc:523] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:55.743721 31174 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:55.751441 31331 ts_tablet_manager.cc:902] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:55.756011 27395 consensus_queue.cc:237] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:55.767419 29336 raft_consensus.cc:1203] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:55.781232 31331 tablet_copy_client.cc:308] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:55.782096 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 9721fc9afa4741b78823e293514e6265 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:51531)
I20240501 20:11:55.782330 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 9721fc9afa4741b78823e293514e6265 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:51531: session id = d56cbcb06b9e418cb617de6a34b946b0-9721fc9afa4741b78823e293514e6265
I20240501 20:11:55.786332 27545 tablet_copy_source_session.cc:205] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:55.789251 29336 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:55.790200 30831 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:55.795120 31331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9721fc9afa4741b78823e293514e6265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:55.807303 31331 tablet_copy_client.cc:783] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:55.807652 31331 tablet_copy_client.cc:647] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:55.809551 27524 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:55.809597 29203 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:55.810380 30831 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:55.811233 30711 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:55.818397 31331 tablet_copy_client.cc:523] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:55.843490 27266 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:55.847002 31331 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:55.844655 31353 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:55.859968 30711 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:55.863761 27525 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:55.882571 26985 catalog_manager.cc:5484] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 5 to 6, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:55.884747 29336 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:55.886180 29203 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:55.915781 27266 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
W20240501 20:11:55.930259 27331 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57b4bd50bc8f4d50957bd0280224c420. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:55.943051 27396 consensus_queue.cc:237] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
W20240501 20:11:55.982470 27332 consensus_peers.cc:485] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 553d443dfb344b2690c260d207901ff1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:55.985814 27133 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:55.986680 31353 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:56.007902 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:56.032400 27266 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:56.033123 31353 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:56.041090 31372 log.cc:924] Time spent T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240501 20:11:56.053572 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:56.057933 29070 raft_consensus.cc:1203] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:56.060963 30711 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:56.071831 26985 catalog_manager.cc:5484] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:56.075318 27133 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:56.086819 27266 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:56.101138 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:56.103691 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:56.104643 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:56.108119 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:56.134454 31403 ts_tablet_manager.cc:902] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:56.142030 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:56.175014 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:56.191970 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:56.195315 31403 tablet_copy_client.cc:308] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:56.196250 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 5272c34458af43bebb5ba5baefac7137 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:55267)
I20240501 20:11:56.196409 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 5272c34458af43bebb5ba5baefac7137 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:55267: session id = d56cbcb06b9e418cb617de6a34b946b0-5272c34458af43bebb5ba5baefac7137
I20240501 20:11:56.200532 27154 tablet_copy_source_session.cc:205] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:56.202999 31403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5272c34458af43bebb5ba5baefac7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:56.216948 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:56.237751 31403 tablet_copy_client.cc:783] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:56.238046 31403 tablet_copy_client.cc:647] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:56.245935 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:56.253069 31403 tablet_copy_client.cc:523] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:56.258414 31403 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:56.275806 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:56.281904 31418 ts_tablet_manager.cc:902] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:56.318203 31331 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:56.319202 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:56.333709 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:56.350072 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:56.359148 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:56.401274 31418 tablet_copy_client.cc:308] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:56.432565 31441 ts_tablet_manager.cc:902] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:56.441325 31331 tablet_bootstrap.cc:492] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:56.441697 31331 ts_tablet_manager.cc:1367] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.595s user 0.063s sys 0.008s
I20240501 20:11:56.442571 31331 raft_consensus.cc:348] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:56.442879 31331 raft_consensus.cc:729] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:56.443351 31331 consensus_queue.cc:260] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:56.460978 31331 ts_tablet_manager.cc:1398] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.019s user 0.005s sys 0.004s
I20240501 20:11:56.462566 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-9721fc9afa4741b78823e293514e6265 received from {username='slave'} at 127.14.106.198:51531
I20240501 20:11:56.462739 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-9721fc9afa4741b78823e293514e6265 on tablet 9721fc9afa4741b78823e293514e6265 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:56.469364 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet 57b4bd50bc8f4d50957bd0280224c420 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:35903)
I20240501 20:11:56.469558 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet 57b4bd50bc8f4d50957bd0280224c420 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:35903: session id = e957d29835544ac587917cc067fb7423-57b4bd50bc8f4d50957bd0280224c420
I20240501 20:11:56.472417 27416 tablet_copy_source_session.cc:205] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:56.475461 31418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b4bd50bc8f4d50957bd0280224c420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:56.495128 31418 tablet_copy_client.cc:783] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:56.497290 31418 tablet_copy_client.cc:647] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:56.500959 31418 tablet_copy_client.cc:523] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:56.512398 31418 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:56.538024 31174 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:56.540215 31441 tablet_copy_client.cc:308] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:56.541383 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet 553d443dfb344b2690c260d207901ff1 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:33893)
I20240501 20:11:56.541549 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet 553d443dfb344b2690c260d207901ff1 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:33893: session id = 565cade0058540a4aa18d7c7a25affbe-553d443dfb344b2690c260d207901ff1
I20240501 20:11:56.545138 27416 tablet_copy_source_session.cc:205] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:56.547536 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d443dfb344b2690c260d207901ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:56.551069 31174 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:56.551606 31174 ts_tablet_manager.cc:1367] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.808s user 0.099s sys 0.060s
I20240501 20:11:56.552387 31174 raft_consensus.cc:348] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:56.552615 31174 raft_consensus.cc:729] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:56.552882 31174 consensus_queue.cc:260] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:56.571498 31174 ts_tablet_manager.cc:1398] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.020s user 0.000s sys 0.007s
I20240501 20:11:56.572305 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-00e3777032eb4d9c80f47855e5f9f87a received from {username='slave'} at 127.14.106.199:37233
I20240501 20:11:56.572576 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-00e3777032eb4d9c80f47855e5f9f87a on tablet 00e3777032eb4d9c80f47855e5f9f87a with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:56.598280 31441 tablet_copy_client.cc:783] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:56.599797 27133 consensus_queue.cc:237] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:56.608358 31441 tablet_copy_client.cc:647] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:56.614713 31441 tablet_copy_client.cc:523] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:56.619985 31441 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
W20240501 20:11:56.625859 27068 consensus_peers.cc:485] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f8902b7b0b649edb10f362713ad36fe. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:56.628536 27396 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:56.629256 31064 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:56.636242 27266 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:56.636893 31064 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:56.646255 31418 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:56.646903 31418 tablet_bootstrap.cc:492] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:56.647264 31418 ts_tablet_manager.cc:1367] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.135s user 0.058s sys 0.000s
I20240501 20:11:56.648290 31418 raft_consensus.cc:348] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:56.648684 31418 raft_consensus.cc:729] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:11:56.648996 31418 consensus_queue.cc:260] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:56.628204 29070 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:56.659046 31064 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:56.669173 31064 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:56.671618 31418 ts_tablet_manager.cc:1398] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.024s user 0.011s sys 0.009s
I20240501 20:11:56.671334 27395 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:56.675375 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session e957d29835544ac587917cc067fb7423-57b4bd50bc8f4d50957bd0280224c420 received from {username='slave'} at 127.14.106.200:35903
I20240501 20:11:56.675523 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session e957d29835544ac587917cc067fb7423-57b4bd50bc8f4d50957bd0280224c420 on tablet 57b4bd50bc8f4d50957bd0280224c420 with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:56.679998 26987 catalog_manager.cc:5484] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:56.683089 27266 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:56.699295 29203 raft_consensus.cc:1203] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:56.703886 29070 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:56.728964 31403 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:56.729612 31403 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:56.729967 31403 ts_tablet_manager.cc:1367] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.472s user 0.055s sys 0.003s
I20240501 20:11:56.731005 31403 raft_consensus.cc:348] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:56.731336 31403 raft_consensus.cc:729] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:56.733486 31403 consensus_queue.cc:260] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:56.734234 31403 ts_tablet_manager.cc:1398] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:11:56.735142 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-5272c34458af43bebb5ba5baefac7137 received from {username='slave'} at 127.14.106.198:55267
I20240501 20:11:56.735287 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-5272c34458af43bebb5ba5baefac7137 on tablet 5272c34458af43bebb5ba5baefac7137 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:56.736644 29336 raft_consensus.cc:1203] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:56.748220 27266 consensus_queue.cc:237] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:56.751760 27133 raft_consensus.cc:1261] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:56.752346 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5b9f1a087114c82b010bd10e4f676c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:56.753054 27524 raft_consensus.cc:1261] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:56.753988 30492 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:56.755867 30492 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:11:56.791472 31499 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:56.795832 27135 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:56.796388 31441 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:56.797091 31441 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:56.797384 31441 ts_tablet_manager.cc:1367] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.177s user 0.043s sys 0.006s
I20240501 20:11:56.798276 31441 raft_consensus.cc:348] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:56.798581 31441 raft_consensus.cc:729] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:56.798910 31441 consensus_queue.cc:260] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:56.799664 31441 ts_tablet_manager.cc:1398] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:56.800838 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-553d443dfb344b2690c260d207901ff1 received from {username='slave'} at 127.14.106.197:33893
I20240501 20:11:56.801002 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-553d443dfb344b2690c260d207901ff1 on tablet 553d443dfb344b2690c260d207901ff1 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:56.803295 26993 catalog_manager.cc:5484] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:56.816452 27524 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:56.822723 29466 raft_consensus.cc:1203] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 20:11:56.840226 29070 raft_consensus.cc:1203] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:56.979112 29203 raft_consensus.cc:1203] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:57.030891 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:57.053756 31174 ts_tablet_manager.cc:902] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:57.058588 27133 consensus_queue.cc:237] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:57.067881 27396 raft_consensus.cc:1261] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:57.068683 31064 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:57.069662 27266 raft_consensus.cc:1261] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:57.070070 27070 consensus_peers.cc:485] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50bf9058852a4ea49733326057d5ff41. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:57.070964 31064 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:57.084192 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:57.094152 31064 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:57.116737 27396 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:57.124920 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:57.125577 26985 catalog_manager.cc:5484] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:57.125957 27266 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:57.154011 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:57.156824 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:57.207883 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:57.245647 31174 tablet_copy_client.cc:308] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:57.246452 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 3f8902b7b0b649edb10f362713ad36fe from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:42311)
I20240501 20:11:57.246690 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 3f8902b7b0b649edb10f362713ad36fe from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:42311: session id = 212de34e3aa24fa1be12c7421df985fc-3f8902b7b0b649edb10f362713ad36fe
I20240501 20:11:57.248901 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:57.249637 27154 tablet_copy_source_session.cc:205] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:57.251106 31174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8902b7b0b649edb10f362713ad36fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:57.257797 31174 tablet_copy_client.cc:783] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:57.258119 31174 tablet_copy_client.cc:647] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:57.260438 31174 tablet_copy_client.cc:523] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:57.264622 31174 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:57.306804 31607 ts_tablet_manager.cc:902] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:11:57.329921 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:57.410836 31607 tablet_copy_client.cc:308] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:46281
I20240501 20:11:57.411751 27286 tablet_copy_service.cc:140] P d8b307156e1346bd9eecf655f49924c7: Received BeginTabletCopySession request for tablet e5b9f1a087114c82b010bd10e4f676c1 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:55097)
I20240501 20:11:57.411964 27286 tablet_copy_service.cc:161] P d8b307156e1346bd9eecf655f49924c7: Beginning new tablet copy session on tablet e5b9f1a087114c82b010bd10e4f676c1 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:55097: session id = d56cbcb06b9e418cb617de6a34b946b0-e5b9f1a087114c82b010bd10e4f676c1
I20240501 20:11:57.416039 27286 tablet_copy_source_session.cc:205] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:57.419389 31607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b9f1a087114c82b010bd10e4f676c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:57.438676 27135 consensus_queue.cc:237] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
W20240501 20:11:57.470360 27071 consensus_peers.cc:485] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2432c8e8e8847fa977055be919e79d6. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:57.470494 27265 raft_consensus.cc:1261] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:57.471571 27524 raft_consensus.cc:1261] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:57.472127 31064 consensus_queue.cc:1035] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:57.473181 31064 consensus_queue.cc:1035] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:57.476156 31607 tablet_copy_client.cc:783] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:57.476455 31607 tablet_copy_client.cc:647] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:57.481971 31607 tablet_copy_client.cc:523] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:57.501243 27135 consensus_queue.cc:237] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:57.521785 31607 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:57.524933 30307 raft_consensus.cc:2932] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
W20240501 20:11:57.529723 27068 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5272c34458af43bebb5ba5baefac7137. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:57.530241 27525 raft_consensus.cc:2932] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:57.533738 29203 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:57.534415 27526 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:57.534619 31064 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:57.534791 27265 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:57.535336 31064 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:57.538378 26994 catalog_manager.cc:5484] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:57.535584 31646 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20240501 20:11:57.575750 31064 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:57.578150 29203 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:57.617192 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:57.638509 27266 raft_consensus.cc:2932] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:57.642582 27524 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:57.644956 26994 catalog_manager.cc:5484] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 reported cstate change: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:57.661581 31667 ts_tablet_manager.cc:902] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:57.672535 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:57.680925 27266 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:57.686527 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:57.691552 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:57.738494 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:57.739689 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:57.776691 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:57.777671 31667 tablet_copy_client.cc:308] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:57.779255 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 50bf9058852a4ea49733326057d5ff41 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:43137)
I20240501 20:11:57.779398 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 50bf9058852a4ea49733326057d5ff41 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:43137: session id = e957d29835544ac587917cc067fb7423-50bf9058852a4ea49733326057d5ff41
I20240501 20:11:57.782415 27154 tablet_copy_source_session.cc:205] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:57.786659 31667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bf9058852a4ea49733326057d5ff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:57.813102 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:57.852890 31667 tablet_copy_client.cc:783] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:57.853309 31667 tablet_copy_client.cc:647] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:57.878466 31667 tablet_copy_client.cc:523] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:57.882481 31667 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:57.930568 31174 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:57.931119 31174 tablet_bootstrap.cc:492] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:57.931437 31174 ts_tablet_manager.cc:1367] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.667s user 0.064s sys 0.025s
I20240501 20:11:57.932262 31174 raft_consensus.cc:348] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:57.932550 31174 raft_consensus.cc:729] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:57.952222 31174 consensus_queue.cc:260] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:57.980995 31174 ts_tablet_manager.cc:1398] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20240501 20:11:57.983436 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-3f8902b7b0b649edb10f362713ad36fe received from {username='slave'} at 127.14.106.199:42311
I20240501 20:11:58.000133 31722 ts_tablet_manager.cc:902] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:58.001056 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-3f8902b7b0b649edb10f362713ad36fe on tablet 3f8902b7b0b649edb10f362713ad36fe with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:58.028192 29335 raft_consensus.cc:1203] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:58.042052 27524 consensus_queue.cc:237] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:58.045492 31607 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:58.045987 31607 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:58.046303 31607 ts_tablet_manager.cc:1367] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.525s user 0.062s sys 0.001s
I20240501 20:11:58.047168 31607 raft_consensus.cc:348] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:58.047432 31607 raft_consensus.cc:729] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:58.047659 31607 consensus_queue.cc:260] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:58.050010 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:58.051043 31607 ts_tablet_manager.cc:1398] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:11:58.054464 27286 tablet_copy_service.cc:342] P d8b307156e1346bd9eecf655f49924c7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-e5b9f1a087114c82b010bd10e4f676c1 received from {username='slave'} at 127.14.106.198:55097
I20240501 20:11:58.054603 27286 tablet_copy_service.cc:433] P d8b307156e1346bd9eecf655f49924c7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-e5b9f1a087114c82b010bd10e4f676c1 on tablet e5b9f1a087114c82b010bd10e4f676c1 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:58.064524 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:58.065384 31740 ts_tablet_manager.cc:902] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
W20240501 20:11:58.096702 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c836bb145a54160966c475e1490da38. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:58.099192 27266 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:58.100567 31157 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:58.115090 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:58.118827 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:58.124372 27396 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:58.125059 31157 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:58.126261 27135 consensus_queue.cc:237] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:58.128732 27395 raft_consensus.cc:1261] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:58.129647 31648 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:58.130806 27265 raft_consensus.cc:1261] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:58.132117 31659 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:11:58.134378 27070 consensus_peers.cc:485] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:58.135433 27068 consensus_peers.cc:485] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50bf9058852a4ea49733326057d5ff41. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:58.144937 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:58.197880 31659 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:58.203982 27394 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:58.207317 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:58.206912 30745 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:58.215387 29202 raft_consensus.cc:1203] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:58.215333 27266 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:58.217551 26993 catalog_manager.cc:5484] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:58.242075 27396 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:58.302083 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:11:58.311163 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 86116us
I20240501 20:11:58.318914 27265 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:58.334885 31740 tablet_copy_client.cc:308] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:58.335924 26994 catalog_manager.cc:5484] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:58.337693 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet c2432c8e8e8847fa977055be919e79d6 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:59561)
I20240501 20:11:58.337898 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet c2432c8e8e8847fa977055be919e79d6 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:59561: session id = 565cade0058540a4aa18d7c7a25affbe-c2432c8e8e8847fa977055be919e79d6
I20240501 20:11:58.345836 27154 tablet_copy_source_session.cc:205] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:58.347507 31740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2432c8e8e8847fa977055be919e79d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:58.376595 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:58.404024 31667 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:58.404626 31667 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:58.404979 31667 ts_tablet_manager.cc:1367] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.523s user 0.057s sys 0.005s
I20240501 20:11:58.413300 31740 tablet_copy_client.cc:783] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:58.413581 31740 tablet_copy_client.cc:647] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:58.415639 31740 tablet_copy_client.cc:523] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:58.423288 27396 consensus_queue.cc:237] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:58.425338 31740 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:58.428958 31722 tablet_copy_client.cc:308] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:58.429502 27265 raft_consensus.cc:1261] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:58.430529 27135 raft_consensus.cc:1261] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:58.431401 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 5272c34458af43bebb5ba5baefac7137 from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:42311)
I20240501 20:11:58.431561 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 5272c34458af43bebb5ba5baefac7137 from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:42311: session id = 212de34e3aa24fa1be12c7421df985fc-5272c34458af43bebb5ba5baefac7137
I20240501 20:11:58.431552 31667 raft_consensus.cc:348] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:58.431777 31667 raft_consensus.cc:729] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
W20240501 20:11:58.432039 27331 consensus_peers.cc:485] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebcdab26b4ed45f596282d4805d1bb1f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:58.432356 31780 consensus_queue.cc:1035] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:58.433218 31667 consensus_queue.cc:260] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:58.433797 31780 consensus_queue.cc:1035] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:58.434432 27154 tablet_copy_source_session.cc:205] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:58.438383 31722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5272c34458af43bebb5ba5baefac7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:58.444972 31722 tablet_copy_client.cc:783] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:58.445209 31722 tablet_copy_client.cc:647] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:58.446738 31722 tablet_copy_client.cc:523] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:58.450201 31722 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:58.475797 31780 raft_consensus.cc:2932] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:58.479414 27133 raft_consensus.cc:2932] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:58.490459 27265 raft_consensus.cc:2932] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:58.494287 26993 catalog_manager.cc:5484] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index -1 to 4, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:11:58.518191 31667 ts_tablet_manager.cc:1398] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.113s user 0.021s sys 0.012s
I20240501 20:11:58.519520 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session e957d29835544ac587917cc067fb7423-50bf9058852a4ea49733326057d5ff41 received from {username='slave'} at 127.14.106.200:43137
I20240501 20:11:58.519678 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session e957d29835544ac587917cc067fb7423-50bf9058852a4ea49733326057d5ff41 on tablet 50bf9058852a4ea49733326057d5ff41 with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:58.545351 31722 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:58.550891 31722 tablet_bootstrap.cc:492] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:58.551265 31722 ts_tablet_manager.cc:1367] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.101s user 0.059s sys 0.002s
I20240501 20:11:58.552093 31722 raft_consensus.cc:348] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:58.552443 31722 raft_consensus.cc:729] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:58.552928 31722 consensus_queue.cc:260] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:58.615188 31722 ts_tablet_manager.cc:1398] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.064s user 0.021s sys 0.020s
I20240501 20:11:58.623549 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-5272c34458af43bebb5ba5baefac7137 received from {username='slave'} at 127.14.106.199:42311
I20240501 20:11:58.623700 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-5272c34458af43bebb5ba5baefac7137 on tablet 5272c34458af43bebb5ba5baefac7137 with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:58.669677 29466 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:11:58.696012 31722 ts_tablet_manager.cc:902] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:11:58.702860 31837 ts_tablet_manager.cc:902] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:11:58.719278 31740 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:58.719801 31740 tablet_bootstrap.cc:492] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:58.720156 31740 ts_tablet_manager.cc:1367] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.295s user 0.059s sys 0.001s
I20240501 20:11:58.721153 31740 raft_consensus.cc:348] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:58.721479 31740 raft_consensus.cc:729] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:58.721760 31740 consensus_queue.cc:260] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:58.725510 31740 ts_tablet_manager.cc:1398] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:11:58.726485 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-c2432c8e8e8847fa977055be919e79d6 received from {username='slave'} at 127.14.106.197:59561
I20240501 20:11:58.726647 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-c2432c8e8e8847fa977055be919e79d6 on tablet c2432c8e8e8847fa977055be919e79d6 with peer 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:58.731201 27200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:46843, user_credentials={real_user=slave}} blocked reactor thread for 95950.4us
I20240501 20:11:58.738597 31722 tablet_copy_client.cc:308] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:11:58.739439 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 50bf9058852a4ea49733326057d5ff41 from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:42311)
I20240501 20:11:58.739667 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 50bf9058852a4ea49733326057d5ff41 from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:42311: session id = 212de34e3aa24fa1be12c7421df985fc-50bf9058852a4ea49733326057d5ff41
I20240501 20:11:58.747689 27154 tablet_copy_source_session.cc:205] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:58.749238 31722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bf9058852a4ea49733326057d5ff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:58.755751 31722 tablet_copy_client.cc:783] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:58.755991 31722 tablet_copy_client.cc:647] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:58.757917 31722 tablet_copy_client.cc:523] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:58.761956 31722 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:11:58.789726 31837 tablet_copy_client.cc:308] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:11:58.791461 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 0c836bb145a54160966c475e1490da38 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:51531)
I20240501 20:11:58.791649 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 0c836bb145a54160966c475e1490da38 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:51531: session id = d56cbcb06b9e418cb617de6a34b946b0-0c836bb145a54160966c475e1490da38
I20240501 20:11:58.796196 27545 tablet_copy_source_session.cc:205] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:58.798368 31837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c836bb145a54160966c475e1490da38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:58.803839 27395 consensus_queue.cc:237] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
W20240501 20:11:58.819409 27332 consensus_peers.cc:485] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2160f17e62749dbb41405a895516b06. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:58.820828 27524 raft_consensus.cc:1261] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:58.821627 31780 consensus_queue.cc:1035] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:58.831650 27135 raft_consensus.cc:1261] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:58.832895 31620 consensus_queue.cc:1035] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:58.840360 31667 ts_tablet_manager.cc:902] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:58.863759 29336 raft_consensus.cc:1203] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:58.871044 31722 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:58.871887 31722 tablet_bootstrap.cc:492] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:11:58.872253 31722 ts_tablet_manager.cc:1367] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.110s user 0.066s sys 0.015s
I20240501 20:11:58.873266 31722 raft_consensus.cc:348] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:58.873608 31722 raft_consensus.cc:729] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:11:58.873862 31722 consensus_queue.cc:260] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:11:58.875355 31722 ts_tablet_manager.cc:1398] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:58.876304 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-50bf9058852a4ea49733326057d5ff41 received from {username='slave'} at 127.14.106.199:42311
I20240501 20:11:58.876470 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-50bf9058852a4ea49733326057d5ff41 on tablet 50bf9058852a4ea49733326057d5ff41 with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:58.890769 31620 raft_consensus.cc:2932] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:58.895629 27525 raft_consensus.cc:2932] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:58.903822 26993 catalog_manager.cc:5484] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:58.910357 31667 tablet_copy_client.cc:308] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:58.911682 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet ebcdab26b4ed45f596282d4805d1bb1f from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:35903)
I20240501 20:11:58.911885 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet ebcdab26b4ed45f596282d4805d1bb1f from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:35903: session id = e957d29835544ac587917cc067fb7423-ebcdab26b4ed45f596282d4805d1bb1f
I20240501 20:11:58.915194 27416 tablet_copy_source_session.cc:205] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:58.916841 31667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcdab26b4ed45f596282d4805d1bb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:58.930742 31667 tablet_copy_client.cc:783] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:58.931216 31667 tablet_copy_client.cc:647] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:58.933276 31667 tablet_copy_client.cc:523] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:58.942865 31667 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:11:58.965691 27135 raft_consensus.cc:2932] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:11:58.999820 31837 tablet_copy_client.cc:783] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:59.000737 31837 tablet_copy_client.cc:647] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:59.003957 31837 tablet_copy_client.cc:523] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:59.032727 31837 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:11:59.056409 29069 raft_consensus.cc:1203] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:59.153339 29336 raft_consensus.cc:1203] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:11:59.218385 31667 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:59.218931 31667 tablet_bootstrap.cc:492] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:11:59.219225 31667 ts_tablet_manager.cc:1367] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.276s user 0.052s sys 0.001s
I20240501 20:11:59.220054 31667 raft_consensus.cc:348] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:59.220284 31667 raft_consensus.cc:729] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:11:59.232017 31667 consensus_queue.cc:260] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:59.248097 31968 ts_tablet_manager.cc:902] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:11:59.266835 31968 tablet_copy_client.cc:308] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:11:59.267975 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet d2160f17e62749dbb41405a895516b06 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:33893)
I20240501 20:11:59.268158 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet d2160f17e62749dbb41405a895516b06 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:33893: session id = 565cade0058540a4aa18d7c7a25affbe-d2160f17e62749dbb41405a895516b06
I20240501 20:11:59.272120 27416 tablet_copy_source_session.cc:205] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:59.273634 31968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2160f17e62749dbb41405a895516b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:59.285260 31667 ts_tablet_manager.cc:1398] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.066s user 0.010s sys 0.010s
I20240501 20:11:59.285626 31968 tablet_copy_client.cc:783] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:59.285950 31968 tablet_copy_client.cc:647] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:59.288236 31968 tablet_copy_client.cc:523] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:59.288446 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session e957d29835544ac587917cc067fb7423-ebcdab26b4ed45f596282d4805d1bb1f received from {username='slave'} at 127.14.106.200:35903
I20240501 20:11:59.288656 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session e957d29835544ac587917cc067fb7423-ebcdab26b4ed45f596282d4805d1bb1f on tablet ebcdab26b4ed45f596282d4805d1bb1f with peer e957d29835544ac587917cc067fb7423
I20240501 20:11:59.292443 31968 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:11:59.310226 29466 raft_consensus.cc:1203] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:59.311570 31837 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:59.317389 31837 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:11:59.317750 31837 ts_tablet_manager.cc:1367] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.285s user 0.056s sys 0.014s
I20240501 20:11:59.320412 31837 raft_consensus.cc:348] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:59.320683 31837 raft_consensus.cc:729] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:11:59.320971 31837 consensus_queue.cc:260] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:59.327961 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.343104 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:59.373824 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.392838 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.400292 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:59.400902 31837 ts_tablet_manager.cc:1398] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.083s user 0.009s sys 0.015s
I20240501 20:11:59.414908 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-0c836bb145a54160966c475e1490da38 received from {username='slave'} at 127.14.106.198:51531
I20240501 20:11:59.415074 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-0c836bb145a54160966c475e1490da38 on tablet 0c836bb145a54160966c475e1490da38 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:59.433368 27524 consensus_queue.cc:237] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:11:59.446961 27395 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:59.447590 31157 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:59.455145 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 944d354124664bc0870f251d8a0831c5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:59.456238 27266 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:59.456888 31157 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:59.477841 30745 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:59.478992 27265 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:59.502948 26985 catalog_manager.cc:5484] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:59.517279 27395 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:11:59.555169 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:59.561179 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:59.586997 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.599169 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.622298 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:11:59.636296 29202 raft_consensus.cc:1203] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:59.664137 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:59.683562 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:59.694270 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.694494 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.730579 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:59.808558 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.840703 31968 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:59.841210 31968 tablet_bootstrap.cc:492] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:11:59.841509 31968 ts_tablet_manager.cc:1367] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.549s user 0.048s sys 0.006s
I20240501 20:11:59.846359 31968 raft_consensus.cc:348] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:59.846639 31968 raft_consensus.cc:729] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:11:59.846936 31968 consensus_queue.cc:260] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:11:59.881481 31968 ts_tablet_manager.cc:1398] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.040s user 0.011s sys 0.009s
I20240501 20:11:59.882311 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-d2160f17e62749dbb41405a895516b06 received from {username='slave'} at 127.14.106.197:33893
I20240501 20:11:59.882467 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-d2160f17e62749dbb41405a895516b06 on tablet d2160f17e62749dbb41405a895516b06 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:59.886533 27524 consensus_queue.cc:237] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
W20240501 20:11:59.905556 27462 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c836bb145a54160966c475e1490da38. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:59.906291 29202 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:59.907161 27266 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:59.908742 32004 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:59.912955 27394 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:59.914907 30745 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:59.915530 30745 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:00.027282 32068 ts_tablet_manager.cc:902] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
W20240501 20:12:00.029196 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 73785.7us
I20240501 20:12:00.029646 30745 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:00.049825 27394 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:00.056710 27266 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
W20240501 20:12:00.088735 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:46293, user_credentials={real_user=slave}} blocked reactor thread for 54163.4us
I20240501 20:12:00.088806 29202 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:00.094902 26993 catalog_manager.cc:5484] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 reported cstate change: config changed from index 4 to 5, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:00.138023 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.152077 29069 raft_consensus.cc:1203] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:12:00.169698 32068 tablet_copy_client.cc:308] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:12:00.189790 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 944d354124664bc0870f251d8a0831c5 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:51531)
I20240501 20:12:00.190038 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 944d354124664bc0870f251d8a0831c5 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:51531: session id = d56cbcb06b9e418cb617de6a34b946b0-944d354124664bc0870f251d8a0831c5
I20240501 20:12:00.193058 27545 tablet_copy_source_session.cc:205] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:00.200047 32068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d354124664bc0870f251d8a0831c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:00.210515 32068 tablet_copy_client.cc:783] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:00.210798 32068 tablet_copy_client.cc:647] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:00.212949 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.216521 32068 tablet_copy_client.cc:523] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:00.220664 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.234827 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:00.238481 32068 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:12:00.242094 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:00.287051 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.306437 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:00.349633 31968 ts_tablet_manager.cc:902] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:12:00.349712 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:00.360827 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.360855 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.371698 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.388429 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:00.402169 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:00.432166 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.460372 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:00.465515 32068 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:00.466158 32068 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:12:00.466614 32068 ts_tablet_manager.cc:1367] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.228s user 0.064s sys 0.000s
I20240501 20:12:00.467588 32068 raft_consensus.cc:348] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:00.467927 32068 raft_consensus.cc:729] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:12:00.468237 32068 consensus_queue.cc:260] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:00.470808 32068 ts_tablet_manager.cc:1398] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:12:00.473726 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-944d354124664bc0870f251d8a0831c5 received from {username='slave'} at 127.14.106.198:51531
I20240501 20:12:00.473847 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-944d354124664bc0870f251d8a0831c5 on tablet 944d354124664bc0870f251d8a0831c5 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:12:00.495327 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:00.603760 27524 consensus_queue.cc:237] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:00.605012 31968 tablet_copy_client.cc:308] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:12:00.605851 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 0c836bb145a54160966c475e1490da38 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:54863)
I20240501 20:12:00.606009 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 0c836bb145a54160966c475e1490da38 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:54863: session id = 565cade0058540a4aa18d7c7a25affbe-0c836bb145a54160966c475e1490da38
I20240501 20:12:00.609231 27545 tablet_copy_source_session.cc:205] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 20:12:00.615052 27460 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17e69dbd245e4b8db397aaffc9145c9f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:00.615993 27266 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:00.616638 32077 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:00.619961 27394 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:00.620997 32080 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:00.624190 31968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c836bb145a54160966c475e1490da38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:00.638868 31968 tablet_copy_client.cc:783] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:00.639228 31968 tablet_copy_client.cc:647] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:00.675540 32077 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:00.681923 27394 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:00.682734 31968 tablet_copy_client.cc:523] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:00.686074 26994 catalog_manager.cc:5484] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:00.686935 27265 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:00.709061 31968 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:12:00.873765 29202 raft_consensus.cc:1203] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:12:01.065335 32237 ts_tablet_manager.cc:902] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:12:01.099257 31968 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:01.102824 32237 tablet_copy_client.cc:308] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:12:01.103788 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 17e69dbd245e4b8db397aaffc9145c9f from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:51531)
I20240501 20:12:01.104002 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 17e69dbd245e4b8db397aaffc9145c9f from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:51531: session id = d56cbcb06b9e418cb617de6a34b946b0-17e69dbd245e4b8db397aaffc9145c9f
I20240501 20:12:01.105183 31968 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:12:01.111135 27545 tablet_copy_source_session.cc:205] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:01.120559 31968 ts_tablet_manager.cc:1367] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.412s user 0.084s sys 0.039s
I20240501 20:12:01.123348 32237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e69dbd245e4b8db397aaffc9145c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:01.122898 31968 raft_consensus.cc:348] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:01.126482 31968 raft_consensus.cc:729] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:12:01.129782 31968 consensus_queue.cc:260] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:01.162079 31968 ts_tablet_manager.cc:1398] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.041s user 0.004s sys 0.002s
I20240501 20:12:01.164229 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-0c836bb145a54160966c475e1490da38 received from {username='slave'} at 127.14.106.197:54863
I20240501 20:12:01.164372 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-0c836bb145a54160966c475e1490da38 on tablet 0c836bb145a54160966c475e1490da38 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:12:01.183732 32237 tablet_copy_client.cc:783] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:01.184047 32237 tablet_copy_client.cc:647] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:01.185976 32237 tablet_copy_client.cc:523] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:01.192345 32237 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:12:01.289398 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:01.353132 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:01.356958 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:01.360285 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:01.443979 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:01.489228 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:01.495095 29069 raft_consensus.cc:1203] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:01.535127 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:01.560593 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:01.668552 32237 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:01.670766 32237 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:12:01.673498 32237 ts_tablet_manager.cc:1367] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.481s user 0.062s sys 0.001s
I20240501 20:12:01.680791 32237 raft_consensus.cc:348] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:01.683604 32237 raft_consensus.cc:729] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:12:01.684039 32237 consensus_queue.cc:260] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:01.710945 32237 ts_tablet_manager.cc:1398] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240501 20:12:01.711963 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-17e69dbd245e4b8db397aaffc9145c9f received from {username='slave'} at 127.14.106.198:51531
I20240501 20:12:01.712121 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-17e69dbd245e4b8db397aaffc9145c9f on tablet 17e69dbd245e4b8db397aaffc9145c9f with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:12:01.794862 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:01.812214 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:01.836480 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:01.848170 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:01.864344 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:01.866878 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:01.878934 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:01.893083 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.030824 29202 raft_consensus.cc:1203] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 20:12:02.221520 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 48930us
I20240501 20:12:02.338434 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.366804 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.435395 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:02.460917 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:02.501585 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.509135 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:02.526754 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:02.571547 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:02.576670 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:02.585541 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.592479 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:02.608976 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.616020 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.673942 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.703049 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.720249 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:03.078023 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:03.078094 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:03.117452 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:03.122253 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:03.131156 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:03.135479 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:03.182781 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:03.212827 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:03.437677 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:03.469412 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:03.521859 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:03.536643 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:03.580669 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:03.592049 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:03.640142 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:03.683933 27135 consensus_queue.cc:237] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:03.686467 29336 raft_consensus.cc:1261] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:03.687322 32070 consensus_queue.cc:1035] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:03.689059 27266 raft_consensus.cc:1261] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:03.693423 32070 consensus_queue.cc:1035] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:03.699817 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:03.699666 31668 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:03.706842 29336 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:03.715884 27266 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:03.725600 26991 catalog_manager.cc:5484] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:03.820832 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc5794a67374cc388da2edc47cbb113 with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:03.835359 32629 tablet_replica.cc:330] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:03.835870 32629 raft_consensus.cc:2229] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:03.836233 32629 raft_consensus.cc:2258] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:03.837950 32629 ts_tablet_manager.cc:1875] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:03.847970 27266 consensus_queue.cc:237] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:03.850113 32629 ts_tablet_manager.cc:1888] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:03.850307 32629 log.cc:1196] T 4dc5794a67374cc388da2edc47cbb113 P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/4dc5794a67374cc388da2edc47cbb113
I20240501 20:12:03.851491 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet 4dc5794a67374cc388da2edc47cbb113 (table rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]) successfully deleted
I20240501 20:12:03.853196 29202 raft_consensus.cc:1261] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:03.853699 27135 raft_consensus.cc:1261] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:03.854614 32301 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240501 20:12:03.859467 32636 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:03.915369 32301 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:03.923967 27133 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:03.936196 26991 catalog_manager.cc:5484] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:03.942648 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b9f1a087114c82b010bd10e4f676c1 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:03.944059 32648 tablet_replica.cc:330] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:03.944777 32648 raft_consensus.cc:2229] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:03.948734 32648 raft_consensus.cc:2258] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:03.952630 32648 ts_tablet_manager.cc:1875] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:03.970408 29203 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:04.045130 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:04.051139 32648 ts_tablet_manager.cc:1888] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:04.067466 32648 log.cc:1196] T e5b9f1a087114c82b010bd10e4f676c1 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/e5b9f1a087114c82b010bd10e4f676c1
I20240501 20:12:04.069996 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:04.072559 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:04.072922 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet e5b9f1a087114c82b010bd10e4f676c1 (table rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]) successfully deleted
I20240501 20:12:04.073115 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:04.087253 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:04.099251 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:04.146845 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:04.166610 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:04.181365 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:04.203431 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:04.281611 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:04.290854 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:04.304078 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:04.318207 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:04.324643 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:04.394026 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:04.541352 27525 consensus_queue.cc:237] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:04.551620 29202 raft_consensus.cc:1261] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:04.556119 32080 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:04.558765 27394 raft_consensus.cc:1261] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:04.563189 32080 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:04.577304 32080 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:04.583006 29202 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:04.606714 26991 catalog_manager.cc:5484] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:04.666206 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb13c1dce8640b28063daa0132d2896 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:04.667488 27395 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:04.679736 32741 tablet_replica.cc:330] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:04.680483 32741 raft_consensus.cc:2229] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:04.680996 32741 raft_consensus.cc:2258] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:04.682861 32741 ts_tablet_manager.cc:1875] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:04.697309 32741 ts_tablet_manager.cc:1888] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:04.699247 32741 log.cc:1196] T 7bb13c1dce8640b28063daa0132d2896 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/7bb13c1dce8640b28063daa0132d2896
I20240501 20:12:04.700572 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 7bb13c1dce8640b28063daa0132d2896 (table rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]) successfully deleted
I20240501 20:12:04.858072 27525 consensus_queue.cc:237] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:04.868903 29070 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:04.869714 32727 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:04.877210 27395 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:04.877243 29202 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:04.877818 32080 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:04.878227 32393 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:04.900590 32080 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:04.904310 27395 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:04.912119 26991 catalog_manager.cc:5484] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:04.916357 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c836bb145a54160966c475e1490da38 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:04.917954 32741 tablet_replica.cc:330] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:04.918174 32741 raft_consensus.cc:2229] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:04.918494 32741 raft_consensus.cc:2258] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:04.919378 32741 ts_tablet_manager.cc:1875] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:04.937032 29202 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:04.975059 32741 ts_tablet_manager.cc:1888] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:04.975353 32741 log.cc:1196] T 0c836bb145a54160966c475e1490da38 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/0c836bb145a54160966c475e1490da38
I20240501 20:12:04.976449 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 0c836bb145a54160966c475e1490da38 (table rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]) successfully deleted
I20240501 20:12:04.977553 29070 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:05.089555 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:05.094048 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:05.108953 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.121655 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.135838 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:05.146288 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.156221 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:05.171416 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.394457 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.457118 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:05.499819 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:05.506752 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.517978 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:05.549782 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:05.562155 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.563652 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.695299 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.717012 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.740295 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:05.746331 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:05.772562 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:05.781003 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:05.801658 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:05.834663 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.016821 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.019080 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.060675 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:06.061796 27525 consensus_queue.cc:237] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:06.065265 27266 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:06.065320 29070 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:06.065876 32080 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:06.066270 32393 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:06.069029 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:06.071004 32080 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:06.076588 29070 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:06.092689 27265 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:06.099819 26985 catalog_manager.cc:5484] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:06.117506 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:06.196682 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.197654 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.208859 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:06.246758 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c414c2eed045bda9d85fe5c2fea654 with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:06.267369 514 tablet_replica.cc:330] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:06.270311 514 raft_consensus.cc:2229] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:06.270227 27525 consensus_queue.cc:237] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:06.278074 514 raft_consensus.cc:2258] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:06.281708 514 ts_tablet_manager.cc:1875] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:06.305833 27135 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:06.304620 29335 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:06.307305 32080 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:06.310598 32393 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:12:06.324702 32393 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.330940 27135 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.337402 29335 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.352655 26988 catalog_manager.cc:5484] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.359474 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f03e02d13f44d188e5eb2774771a0d with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:06.392755 540 tablet_replica.cc:330] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:06.393270 540 raft_consensus.cc:2229] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:06.393883 540 raft_consensus.cc:2258] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:06.395267 540 ts_tablet_manager.cc:1875] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:06.418057 514 ts_tablet_manager.cc:1888] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:06.418916 514 log.cc:1196] T 74c414c2eed045bda9d85fe5c2fea654 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/74c414c2eed045bda9d85fe5c2fea654
I20240501 20:12:06.420572 540 ts_tablet_manager.cc:1888] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:06.420748 540 log.cc:1196] T e7f03e02d13f44d188e5eb2774771a0d P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/e7f03e02d13f44d188e5eb2774771a0d
I20240501 20:12:06.425217 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet e7f03e02d13f44d188e5eb2774771a0d (table rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]) successfully deleted
I20240501 20:12:06.443236 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet 74c414c2eed045bda9d85fe5c2fea654 (table rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]) successfully deleted
I20240501 20:12:06.493675 27525 consensus_queue.cc:237] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:06.506559 29335 raft_consensus.cc:1261] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:06.506644 27394 raft_consensus.cc:1261] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:06.507308 32393 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:06.507898 32080 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:06.530222 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.540216 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:06.557487 32080 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.560546 29335 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.563320 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:06.572685 26985 catalog_manager.cc:5484] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc reported cstate change: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.580426 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet 00e3777032eb4d9c80f47855e5f9f87a with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:06.580792 514 tablet_replica.cc:330] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:06.581003 514 raft_consensus.cc:2229] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:06.581238 514 raft_consensus.cc:2258] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:06.582043 514 ts_tablet_manager.cc:1875] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:06.584663 27395 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.593838 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.613231 514 ts_tablet_manager.cc:1888] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:06.624951 514 log.cc:1196] T 00e3777032eb4d9c80f47855e5f9f87a P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/00e3777032eb4d9c80f47855e5f9f87a
I20240501 20:12:06.625917 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet 00e3777032eb4d9c80f47855e5f9f87a (table rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]) successfully deleted
I20240501 20:12:06.663223 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.673769 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:06.705281 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:06.727064 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:06.896631 27524 consensus_queue.cc:237] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:06.922837 29335 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:06.924300 32393 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:06.930259 27135 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:06.932340 571 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:06.953140 571 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.962249 27135 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.965279 26985 catalog_manager.cc:5484] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:06.973343 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet f034a679870b44caa238f04a8b49d302 with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:07.010105 649 tablet_replica.cc:330] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:07.010409 649 raft_consensus.cc:2229] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:07.010841 649 raft_consensus.cc:2258] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:07.014689 649 ts_tablet_manager.cc:1875] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:07.015120 29335 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:07.026708 649 ts_tablet_manager.cc:1888] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:07.026903 649 log.cc:1196] T f034a679870b44caa238f04a8b49d302 P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/f034a679870b44caa238f04a8b49d302
I20240501 20:12:07.027680 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet f034a679870b44caa238f04a8b49d302 (table rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]) successfully deleted
I20240501 20:12:07.176348 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.199509 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:07.234025 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:07.252221 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:07.265314 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.292052 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.295189 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.319521 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:07.410914 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.455430 27395 consensus_queue.cc:237] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:07.473098 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:07.512833 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.517427 29467 raft_consensus.cc:1261] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:07.517468 27266 raft_consensus.cc:1261] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:07.518222 621 consensus_queue.cc:1035] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:07.518923 621 consensus_queue.cc:1035] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:07.536342 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:07.546653 32600 raft_consensus.cc:2932] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:07.553751 27266 raft_consensus.cc:2932] T ebcdab26b4ed45f596282d4805d1bb1f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:07.557025 26987 catalog_manager.cc:5484] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:07.578584 29467 raft_consensus.cc:2932] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:07.601791 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:07.609923 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:07.621485 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.639102 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet ebcdab26b4ed45f596282d4805d1bb1f with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:07.647169 756 tablet_replica.cc:330] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:07.647696 756 raft_consensus.cc:2229] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:07.648030 756 raft_consensus.cc:2258] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:07.648820 756 ts_tablet_manager.cc:1875] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:07.664306 756 ts_tablet_manager.cc:1888] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:07.678997 756 log.cc:1196] T ebcdab26b4ed45f596282d4805d1bb1f P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/ebcdab26b4ed45f596282d4805d1bb1f
I20240501 20:12:07.679930 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet ebcdab26b4ed45f596282d4805d1bb1f (table rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]) successfully deleted
I20240501 20:12:07.688129 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.728112 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.778661 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.788535 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:07.799607 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:07.838549 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:07.850198 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:07.855334 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:07.949999 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:08.176592 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:08.182031 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:08.222327 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:08.271023 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:08.291124 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:08.292232 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:08.379410 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:08.387363 27524 consensus_queue.cc:237] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:08.415814 29069 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:08.416623 32080 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:08.424376 29202 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:08.425079 32080 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:08.427027 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:08.433578 32080 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:08.436051 29070 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:08.436410 29203 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:08.455884 26991 catalog_manager.cc:5484] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 6 to 7, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:08.542445 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c836bb145a54160966c475e1490da38 with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:08.543141 888 tablet_replica.cc:330] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:08.543424 888 raft_consensus.cc:2229] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:08.543956 888 raft_consensus.cc:2258] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:08.544775 888 ts_tablet_manager.cc:1875] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:08.548415 27395 consensus_queue.cc:237] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:08.552093 888 ts_tablet_manager.cc:1888] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:08.552425 888 log.cc:1196] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/0c836bb145a54160966c475e1490da38
I20240501 20:12:08.554174 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet 0c836bb145a54160966c475e1490da38 (table rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]) successfully deleted
I20240501 20:12:08.577355 29467 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:08.578186 32600 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:08.585716 29335 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:08.586524 32600 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:08.596637 29203 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:08.597646 732 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:08.608191 27265 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:08.608857 32600 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:08.616881 32600 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:08.622252 29336 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:08.629360 29467 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:08.631057 26988 catalog_manager.cc:5484] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:08.637178 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b4bd50bc8f4d50957bd0280224c420 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:08.647732 900 tablet_replica.cc:330] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:08.647948 900 raft_consensus.cc:2229] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:08.647557 29202 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:08.650897 900 raft_consensus.cc:2258] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:08.652582 900 ts_tablet_manager.cc:1875] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:08.666318 27265 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:08.675587 900 ts_tablet_manager.cc:1888] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:08.683792 900 log.cc:1196] T 57b4bd50bc8f4d50957bd0280224c420 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/57b4bd50bc8f4d50957bd0280224c420
I20240501 20:12:08.685096 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet 57b4bd50bc8f4d50957bd0280224c420 (table rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]) successfully deleted
I20240501 20:12:08.785641 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:08.829418 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:08.859640 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:08.861151 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:08.886433 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:08.894668 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:08.942466 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:08.973001 27524 consensus_queue.cc:237] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:08.981246 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.002358 27133 raft_consensus.cc:1261] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:09.002364 29070 raft_consensus.cc:1261] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:09.002935 571 consensus_queue.cc:1035] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:09.003520 875 consensus_queue.cc:1035] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:09.037490 875 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:09.045820 27133 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:09.046875 26987 catalog_manager.cc:5484] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:09.051226 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet f02a619bc1584d53ac902c8d8baff976 with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:09.051649 888 tablet_replica.cc:330] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:09.051940 888 raft_consensus.cc:2229] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:09.052318 888 raft_consensus.cc:2258] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:09.053123 888 ts_tablet_manager.cc:1875] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:09.061400 888 ts_tablet_manager.cc:1888] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:09.062872 888 log.cc:1196] T f02a619bc1584d53ac902c8d8baff976 P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/f02a619bc1584d53ac902c8d8baff976
I20240501 20:12:09.063871 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet f02a619bc1584d53ac902c8d8baff976 (table rebalance_test_table_0 [id=565e816257734920a19d1af631224041]) successfully deleted
I20240501 20:12:09.085093 29069 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:09.188467 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.227110 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:09.247418 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.248178 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:09.293344 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.384853 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.403116 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:09.426959 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:09.540771 27395 consensus_queue.cc:237] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:09.588117 29069 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:09.589269 27133 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:09.590288 32600 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:09.591071 32600 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:09.597453 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:09.631211 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.642766 732 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:09.647631 29069 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:09.652485 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.655740 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:09.674273 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:46293, user_credentials={real_user=slave}} blocked reactor thread for 84149.2us
I20240501 20:12:09.695567 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:09.705055 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.731415 27133 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:09.738075 26989 catalog_manager.cc:5484] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:09.747388 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:09.766788 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:09.778142 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:09.805104 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.808632 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:09.911415 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:09.916977 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.927430 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 553d443dfb344b2690c260d207901ff1 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:09.928184 1102 tablet_replica.cc:330] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:09.928398 1102 raft_consensus.cc:2229] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:09.928658 1102 raft_consensus.cc:2258] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:09.929399 1102 ts_tablet_manager.cc:1875] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:09.945977 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.952354 1102 ts_tablet_manager.cc:1888] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:09.952535 1102 log.cc:1196] T 553d443dfb344b2690c260d207901ff1 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/553d443dfb344b2690c260d207901ff1
I20240501 20:12:09.960554 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 553d443dfb344b2690c260d207901ff1 (table rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]) successfully deleted
I20240501 20:12:09.970062 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:09.993191 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:10.322175 27133 consensus_queue.cc:237] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:10.336886 29335 raft_consensus.cc:1261] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:10.337010 27395 raft_consensus.cc:1261] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:10.337529 1008 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:10.337980 1087 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:10.338497 29466 raft_consensus.cc:1261] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:10.341365 1087 consensus_queue.cc:1035] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:10.361047 1087 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.367884 29466 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.375835 26988 catalog_manager.cc:5484] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.375762 29336 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.379832 27395 raft_consensus.cc:2932] T 50bf9058852a4ea49733326057d5ff41 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.402473 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 50bf9058852a4ea49733326057d5ff41 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:10.403151 1102 tablet_replica.cc:330] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:10.403446 1102 raft_consensus.cc:2229] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:10.403822 1102 raft_consensus.cc:2258] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:10.405061 1102 ts_tablet_manager.cc:1875] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:10.416311 1102 ts_tablet_manager.cc:1888] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:10.416483 1102 log.cc:1196] T 50bf9058852a4ea49733326057d5ff41 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/50bf9058852a4ea49733326057d5ff41
I20240501 20:12:10.417469 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 50bf9058852a4ea49733326057d5ff41 (table rebalance_test_table_0 [id=565e816257734920a19d1af631224041]) successfully deleted
I20240501 20:12:10.425424 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:10.470219 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 50131.6us
I20240501 20:12:10.481406 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:10.485599 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:10.491662 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:10.629797 27524 consensus_queue.cc:237] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:10.639159 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:10.644723 29069 raft_consensus.cc:1261] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:10.645340 1159 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:10.648312 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:10.655822 27395 raft_consensus.cc:1261] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:10.656459 1159 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:10.658330 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:10.681803 875 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:10.688210 29068 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:10.685853 27394 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:10.695727 26988 catalog_manager.cc:5484] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:10.701923 27133 consensus_queue.cc:237] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:10.704593 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c836d9afa0f439b978bc2fe5b5e499d with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:10.704977 1102 tablet_replica.cc:330] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:10.705158 1102 raft_consensus.cc:2229] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:10.705439 1102 raft_consensus.cc:2258] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:10.706179 1102 ts_tablet_manager.cc:1875] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:10.710041 27266 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:10.710685 1008 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:10.711226 29336 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:10.714751 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:10.716890 1085 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:10.717234 29069 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:10.717852 1008 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:10.728904 1102 ts_tablet_manager.cc:1888] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:10.729050 1102 log.cc:1196] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/6c836d9afa0f439b978bc2fe5b5e499d
I20240501 20:12:10.729893 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 6c836d9afa0f439b978bc2fe5b5e499d (table rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]) successfully deleted
I20240501 20:12:10.743122 1085 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.743240 29070 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.745978 29335 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.751068 27266 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.757809 26988 catalog_manager.cc:5484] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:10.769652 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8902b7b0b649edb10f362713ad36fe with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:10.770713 1223 tablet_replica.cc:330] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:10.776371 1223 raft_consensus.cc:2229] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:10.776822 1223 raft_consensus.cc:2258] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:10.777715 1223 ts_tablet_manager.cc:1875] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:10.852084 1223 ts_tablet_manager.cc:1888] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:10.852246 1223 log.cc:1196] T 3f8902b7b0b649edb10f362713ad36fe P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f8902b7b0b649edb10f362713ad36fe
I20240501 20:12:10.853070 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet 3f8902b7b0b649edb10f362713ad36fe (table rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]) successfully deleted
I20240501 20:12:10.961026 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:10.974084 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:10.975108 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:10.993386 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.041623 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.056424 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:11.113703 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:11.131577 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.181762 27266 consensus_queue.cc:237] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:11.193789 29202 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:11.195086 1219 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:11.196789 27524 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:11.197412 1219 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:11.210186 442 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:11.210901 27525 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:11.217231 26988 catalog_manager.cc:5484] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:11.229795 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet 26cb2329d176441ca9127187db2d157a with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:11.254355 1303 tablet_replica.cc:330] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:11.254698 1303 raft_consensus.cc:2229] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:11.255118 1303 raft_consensus.cc:2258] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:11.255939 1303 ts_tablet_manager.cc:1875] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:11.277760 29202 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:11.348238 1303 ts_tablet_manager.cc:1888] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:11.361166 1303 log.cc:1196] T 26cb2329d176441ca9127187db2d157a P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/26cb2329d176441ca9127187db2d157a
I20240501 20:12:11.363864 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet 26cb2329d176441ca9127187db2d157a (table rebalance_test_table_0 [id=565e816257734920a19d1af631224041]) successfully deleted
I20240501 20:12:11.388320 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.433944 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:11.443323 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.503590 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:11.503571 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:11.507448 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.527452 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:11.614821 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.768734 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.810153 27525 consensus_queue.cc:237] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:11.818259 29068 raft_consensus.cc:1261] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:11.819057 875 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:11.820850 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:11.823634 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.832188 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:11.847013 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.864961 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.871090 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:11.874706 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:11.897797 1159 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:11.903590 29069 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:11.961531 26990 catalog_manager.cc:5484] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
W20240501 20:12:11.981395 27330 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:43861, user_credentials={real_user=slave}} blocked reactor thread for 79763.1us
W20240501 20:12:11.983146 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 67097.5us
I20240501 20:12:11.998489 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:12.022310 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:12.048141 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c836d9afa0f439b978bc2fe5b5e499d with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:12.050094 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:12.055261 27524 consensus_queue.cc:237] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:12.057724 29069 raft_consensus.cc:1261] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:12.058286 1159 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:12.060564 1159 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:12.064371 29068 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:12.071503 26988 catalog_manager.cc:5484] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
W20240501 20:12:12.072713 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c836d9afa0f439b978bc2fe5b5e499d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:12.079726 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6c836d9afa0f439b978bc2fe5b5e499d with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:12.081224 1432 tablet_replica.cc:330] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:12.086449 1432 raft_consensus.cc:2229] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:12.086787 1432 raft_consensus.cc:2258] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:12.088376 1432 ts_tablet_manager.cc:1875] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:12.120122 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:12.151216 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:12.159729 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:12.171752 1432 ts_tablet_manager.cc:1888] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:12.177378 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:12.175716 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:12.180141 1432 log.cc:1196] T 6c836d9afa0f439b978bc2fe5b5e499d P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/6c836d9afa0f439b978bc2fe5b5e499d
I20240501 20:12:12.186092 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet 6c836d9afa0f439b978bc2fe5b5e499d (table rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]) successfully deleted
I20240501 20:12:12.442394 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:12.454382 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:12.468667 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:12.473040 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:12.474411 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:12.483777 27524 consensus_queue.cc:237] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:12.493912 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:12.549548 29466 raft_consensus.cc:1261] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:12.550325 1159 consensus_queue.cc:1035] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:12.558344 27266 raft_consensus.cc:1261] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:12.558980 1159 consensus_queue.cc:1035] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:12.561779 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:12.599102 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:12.606292 1159 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:12.611661 29467 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:12.622648 26990 catalog_manager.cc:5484] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:12.664903 27266 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:12.669374 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e88937e1ec49bfbaffe535e445e40a with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:12.669776 1520 ts_tablet_manager.cc:902] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:12:12.697674 1525 tablet_replica.cc:330] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:12.697974 1525 raft_consensus.cc:2229] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:12.698263 1525 raft_consensus.cc:2258] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:12.716100 1525 ts_tablet_manager.cc:1875] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:12.750970 1520 tablet_copy_client.cc:308] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:12:12.751785 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 6c836d9afa0f439b978bc2fe5b5e499d from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:51531)
I20240501 20:12:12.751964 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 6c836d9afa0f439b978bc2fe5b5e499d from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:51531: session id = d56cbcb06b9e418cb617de6a34b946b0-6c836d9afa0f439b978bc2fe5b5e499d
I20240501 20:12:12.755362 27545 tablet_copy_source_session.cc:205] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:12.760728 1520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c836d9afa0f439b978bc2fe5b5e499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.768679 1520 tablet_copy_client.cc:783] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:12.769117 1520 tablet_copy_client.cc:647] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:12.770931 1520 tablet_copy_client.cc:523] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:12.777806 1520 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:12:12.807474 1525 ts_tablet_manager.cc:1888] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:12.807827 1525 log.cc:1196] T c0e88937e1ec49bfbaffe535e445e40a P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0e88937e1ec49bfbaffe535e445e40a
I20240501 20:12:12.811630 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet c0e88937e1ec49bfbaffe535e445e40a (table rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]) successfully deleted
I20240501 20:12:12.818485 27135 consensus_queue.cc:237] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:12.834435 29069 raft_consensus.cc:1261] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:12.835144 27266 raft_consensus.cc:1261] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:12.835811 1008 consensus_queue.cc:1035] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:12.837865 1008 consensus_queue.cc:1035] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:12.867543 1008 raft_consensus.cc:2932] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:12.871471 29068 raft_consensus.cc:2932] T c2432c8e8e8847fa977055be919e79d6 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:12.882004 1520 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:12.882541 1520 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:12:12.882879 1520 ts_tablet_manager.cc:1367] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.105s user 0.067s sys 0.022s
I20240501 20:12:12.883772 1520 raft_consensus.cc:348] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:12.884048 1520 raft_consensus.cc:729] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:12:12.884331 1520 consensus_queue.cc:260] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:12.887879 26990 catalog_manager.cc:5484] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:12.892472 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet c2432c8e8e8847fa977055be919e79d6 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:12.931892 27264 raft_consensus.cc:2932] T c2432c8e8e8847fa977055be919e79d6 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:12.940485 1581 tablet_replica.cc:330] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:12.940873 1581 raft_consensus.cc:2229] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:12.941202 1581 raft_consensus.cc:2258] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:12.942270 1581 ts_tablet_manager.cc:1875] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:12.984218 1520 ts_tablet_manager.cc:1398] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.101s user 0.032s sys 0.057s
I20240501 20:12:12.985586 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-6c836d9afa0f439b978bc2fe5b5e499d received from {username='slave'} at 127.14.106.198:51531
I20240501 20:12:12.985726 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-6c836d9afa0f439b978bc2fe5b5e499d on tablet 6c836d9afa0f439b978bc2fe5b5e499d with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:12:13.015515 1581 ts_tablet_manager.cc:1888] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:13.015650 1581 log.cc:1196] T c2432c8e8e8847fa977055be919e79d6 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/c2432c8e8e8847fa977055be919e79d6
I20240501 20:12:13.022735 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet c2432c8e8e8847fa977055be919e79d6 (table rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]) successfully deleted
I20240501 20:12:13.046828 29202 raft_consensus.cc:1203] T 6c836d9afa0f439b978bc2fe5b5e499d P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:12:13.068145 27504 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:13.079823 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:13.084257 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:13.093227 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:13.095538 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:13.123227 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:13.157646 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:13.185287 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:13.330947 27524 consensus_queue.cc:237] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:13.341105 29202 raft_consensus.cc:1261] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:13.341928 1159 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240501 20:12:13.353585 1420 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:13.361323 29203 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:13.369717 26990 catalog_manager.cc:5484] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 reported cstate change: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:13.378636 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb13c1dce8640b28063daa0132d2896 with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:13.385907 27524 consensus_queue.cc:237] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
W20240501 20:12:13.396450 27461 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bb13c1dce8640b28063daa0132d2896. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:13.397065 29203 raft_consensus.cc:1261] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:13.397562 1652 tablet_replica.cc:330] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:13.397894 1420 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:13.398137 1652 raft_consensus.cc:2229] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:13.398478 1652 raft_consensus.cc:2258] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:13.399458 1652 ts_tablet_manager.cc:1875] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:13.400588 1159 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:13.405468 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7bb13c1dce8640b28063daa0132d2896 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:13.408151 26990 catalog_manager.cc:5484] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:13.415305 29203 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:13.417743 1652 ts_tablet_manager.cc:1888] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:13.417968 1652 log.cc:1196] T 7bb13c1dce8640b28063daa0132d2896 P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/7bb13c1dce8640b28063daa0132d2896
I20240501 20:12:13.418862 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet 7bb13c1dce8640b28063daa0132d2896 (table rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]) successfully deleted
I20240501 20:12:13.733209 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:13.736658 27524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d"
dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55452
I20240501 20:12:13.736939 27524 raft_consensus.cc:595] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:13.738337 27114 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:13.752593 1420 raft_consensus.cc:979] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: : Instructing follower 4d603ea5d17441b3bac562bf08ec83fe to start an election
I20240501 20:12:13.752772 1420 raft_consensus.cc:1067] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Signalling peer 4d603ea5d17441b3bac562bf08ec83fe to start an election
I20240501 20:12:13.753855 27135 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d"
dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
from {username='slave'} at 127.14.106.196:49765
I20240501 20:12:13.754091 27135 raft_consensus.cc:482] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:12:13.754174 27135 raft_consensus.cc:3037] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:13.756958 27135 raft_consensus.cc:504] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:13.765409 27135 leader_election.cc:288] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 2 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
I20240501 20:12:13.766678 27524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:12:13.766880 27524 raft_consensus.cc:3032] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:12:13.766935 27524 raft_consensus.cc:729] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:12:13.767184 27524 consensus_queue.cc:260] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:13.767553 27524 raft_consensus.cc:3037] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:13.773680 29336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f03e02d13f44d188e5eb2774771a0d" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "212de34e3aa24fa1be12c7421df985fc"
I20240501 20:12:13.774004 29336 raft_consensus.cc:3037] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:13.770608 27524 raft_consensus.cc:2454] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 2.
I20240501 20:12:13.777391 27069 leader_election.cc:302] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7; no voters:
I20240501 20:12:13.777794 1635 raft_consensus.cc:2781] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:12:13.777953 1635 raft_consensus.cc:686] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 2 LEADER]: Becoming Leader. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Running, Role: LEADER
I20240501 20:12:13.778302 1635 consensus_queue.cc:237] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:13.784152 29336 raft_consensus.cc:2454] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d603ea5d17441b3bac562bf08ec83fe in term 2.
I20240501 20:12:13.806577 26989 catalog_manager.cc:5484] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: term changed from 1 to 2, leader changed from 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) to 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193). New cstate: current_term: 2 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:13.849517 27504 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:13.931396 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:13.966318 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:13.984724 1742 ts_tablet_manager.cc:902] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:12:13.987365 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.015508 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.023823 1742 tablet_copy_client.cc:308] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:12:14.039235 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.044549 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 7bb13c1dce8640b28063daa0132d2896 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:55441)
I20240501 20:12:14.044703 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 7bb13c1dce8640b28063daa0132d2896 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:55441: session id = e957d29835544ac587917cc067fb7423-7bb13c1dce8640b28063daa0132d2896
I20240501 20:12:14.047881 27545 tablet_copy_source_session.cc:205] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:14.053651 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.053761 1742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb13c1dce8640b28063daa0132d2896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.066529 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:14.076126 27504 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:14.164402 1742 tablet_copy_client.cc:783] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:14.164891 1742 tablet_copy_client.cc:647] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:14.184082 27504 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:14.185425 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:14.186888 1742 tablet_copy_client.cc:523] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:14.191316 1742 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:12:14.200647 27525 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:14.201390 1008 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:14.204739 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.258771 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.278811 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:14.282106 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.290066 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:14.291019 29336 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:14.291690 1635 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:14.332839 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.348304 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.373620 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:14.375984 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:14.378795 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:14.424716 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.645671 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:14.646730 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.726774 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.761801 27504 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:14.762007 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.799865 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:14.824790 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:14.826947 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:14.831820 1742 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:14.832401 1742 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:12:14.839246 1742 ts_tablet_manager.cc:1367] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.648s user 0.055s sys 0.016s
I20240501 20:12:14.844889 1742 raft_consensus.cc:348] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:14.849412 1742 raft_consensus.cc:729] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.851572 1742 consensus_queue.cc:260] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:14.863771 1742 ts_tablet_manager.cc:1398] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 20:12:14.875824 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session e957d29835544ac587917cc067fb7423-7bb13c1dce8640b28063daa0132d2896 received from {username='slave'} at 127.14.106.200:55441
I20240501 20:12:14.875960 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session e957d29835544ac587917cc067fb7423-7bb13c1dce8640b28063daa0132d2896 on tablet 7bb13c1dce8640b28063daa0132d2896 with peer e957d29835544ac587917cc067fb7423
I20240501 20:12:14.917734 27135 consensus_queue.cc:237] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:14.950040 29466 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:14.950163 29068 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:14.950722 1008 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:14.951156 27264 raft_consensus.cc:1261] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:14.953742 1795 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:14.954370 1795 consensus_queue.cc:1035] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:15.001487 1008 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.012719 29466 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.014031 27264 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.029258 26988 catalog_manager.cc:5484] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.058300 29068 raft_consensus.cc:2932] T b5427fbc8a244a69974fd8d0f5f3d08d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.087715 29466 raft_consensus.cc:1203] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:12:15.102911 27395 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "553d443dfb344b2690c260d207901ff1"
dest_uuid: "e677127814314770b3370764d0ff9d06"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52824
I20240501 20:12:15.103252 27395 raft_consensus.cc:595] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:15.148377 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet b5427fbc8a244a69974fd8d0f5f3d08d with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:15.173590 1917 tablet_replica.cc:330] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:15.174289 1917 raft_consensus.cc:2229] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:15.174788 1917 raft_consensus.cc:2258] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:15.176192 1917 ts_tablet_manager.cc:1875] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:15.177990 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:15.191197 1917 ts_tablet_manager.cc:1888] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:15.192623 1917 log.cc:1196] T b5427fbc8a244a69974fd8d0f5f3d08d P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/b5427fbc8a244a69974fd8d0f5f3d08d
I20240501 20:12:15.192732 27135 consensus_queue.cc:237] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:15.194113 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet b5427fbc8a244a69974fd8d0f5f3d08d (table rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]) successfully deleted
I20240501 20:12:15.204608 29466 raft_consensus.cc:1261] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:15.205255 1008 consensus_queue.cc:1035] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:15.213723 27395 raft_consensus.cc:1261] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:15.214718 1876 consensus_queue.cc:1035] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:15.224455 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:15.232514 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:15.235150 1876 raft_consensus.cc:2932] T 309e90bbd9f140a8beca9b3da3f75db4 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.239300 29467 raft_consensus.cc:2932] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.244431 27133 consensus_queue.cc:237] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:15.245018 26985 catalog_manager.cc:5484] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.246152 27395 raft_consensus.cc:2932] T 309e90bbd9f140a8beca9b3da3f75db4 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:15.250803 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 309e90bbd9f140a8beca9b3da3f75db4 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:15.263911 27505 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:15.273540 29336 raft_consensus.cc:1261] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:15.274206 1877 consensus_queue.cc:1035] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:15.269662 1941 tablet_replica.cc:330] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:15.277004 1941 raft_consensus.cc:2229] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:15.279403 1941 raft_consensus.cc:2258] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:15.286077 29050 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:15.286717 1941 ts_tablet_manager.cc:1875] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:12:15.296207 27330 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:43861, user_credentials={real_user=slave}} blocked reactor thread for 95812.7us
I20240501 20:12:15.311486 1876 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:15.312240 1941 ts_tablet_manager.cc:1888] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:15.312351 1941 log.cc:1196] T 309e90bbd9f140a8beca9b3da3f75db4 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/309e90bbd9f140a8beca9b3da3f75db4
I20240501 20:12:15.315857 29335 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:15.317737 26989 catalog_manager.cc:5484] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:15.318872 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet 309e90bbd9f140a8beca9b3da3f75db4 (table rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]) successfully deleted
I20240501 20:12:15.328002 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc5794a67374cc388da2edc47cbb113 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:15.329926 1942 raft_consensus.cc:979] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: : Instructing follower 565cade0058540a4aa18d7c7a25affbe to start an election
I20240501 20:12:15.330077 1942 raft_consensus.cc:1067] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Signalling peer 565cade0058540a4aa18d7c7a25affbe to start an election
I20240501 20:12:15.330981 27135 consensus_queue.cc:237] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:15.332203 29068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "553d443dfb344b2690c260d207901ff1"
dest_uuid: "565cade0058540a4aa18d7c7a25affbe"
from {username='slave'} at 127.14.106.195:52159
I20240501 20:12:15.332535 29068 raft_consensus.cc:482] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:12:15.332764 29068 raft_consensus.cc:3037] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:12:15.335431 27071 consensus_peers.cc:485] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dc5794a67374cc388da2edc47cbb113. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:15.336997 29068 raft_consensus.cc:504] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:15.339140 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "565cade0058540a4aa18d7c7a25affbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:12:15.339345 27135 raft_consensus.cc:3037] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:15.342188 27135 raft_consensus.cc:2454] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 565cade0058540a4aa18d7c7a25affbe in term 2.
I20240501 20:12:15.342842 29005 leader_election.cc:302] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe, 565cade0058540a4aa18d7c7a25affbe; no voters:
I20240501 20:12:15.343261 1890 raft_consensus.cc:2781] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:12:15.344662 29068 leader_election.cc:288] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [CANDIDATE]: Term 2 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:12:15.345248 27395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "565cade0058540a4aa18d7c7a25affbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e677127814314770b3370764d0ff9d06"
I20240501 20:12:15.345469 27395 raft_consensus.cc:3032] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:12:15.345542 27395 raft_consensus.cc:729] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:12:15.345922 27395 consensus_queue.cc:260] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:15.346333 27395 raft_consensus.cc:3037] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:15.349681 29336 raft_consensus.cc:1261] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:15.350241 1008 consensus_queue.cc:1035] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:15.358367 1876 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:15.359131 29334 raft_consensus.cc:2932] T 4dc5794a67374cc388da2edc47cbb113 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:15.362174 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4dc5794a67374cc388da2edc47cbb113 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:15.364805 26989 catalog_manager.cc:5484] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 6 to 7, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:15.374960 27133 raft_consensus.cc:1226] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Rejecting Update request from peer e677127814314770b3370764d0ff9d06 for earlier term 1. Current term is 2. Ops: []
I20240501 20:12:15.381103 27245 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:15.381917 27395 raft_consensus.cc:2454] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 565cade0058540a4aa18d7c7a25affbe in term 2.
I20240501 20:12:15.386575 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:15.398811 1890 raft_consensus.cc:686] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 2 LEADER]: Becoming Leader. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Running, Role: LEADER
I20240501 20:12:15.399528 1890 consensus_queue.cc:237] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:15.419977 26989 catalog_manager.cc:5484] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: term changed from 1 to 2, leader changed from e677127814314770b3370764d0ff9d06 (127.14.106.195) to 565cade0058540a4aa18d7c7a25affbe (127.14.106.197). New cstate: current_term: 2 leader_uuid: "565cade0058540a4aa18d7c7a25affbe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:15.420660 1951 tablet_replica.cc:330] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:15.429900 1951 raft_consensus.cc:2229] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:15.430284 1951 raft_consensus.cc:2258] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:15.434304 1951 ts_tablet_manager.cc:1875] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:15.449893 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:15.466454 1951 ts_tablet_manager.cc:1888] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:15.466601 1951 log.cc:1196] T 4dc5794a67374cc388da2edc47cbb113 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/4dc5794a67374cc388da2edc47cbb113
I20240501 20:12:15.468362 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 4dc5794a67374cc388da2edc47cbb113 (table rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]) successfully deleted
I20240501 20:12:15.783703 27395 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Refusing update from remote peer 565cade0058540a4aa18d7c7a25affbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:15.784579 1890 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:15.820832 27133 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Refusing update from remote peer 565cade0058540a4aa18d7c7a25affbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:15.821542 1955 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:15.830785 2057 ts_tablet_manager.cc:902] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:12:15.916993 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:15.940609 2057 tablet_copy_client.cc:308] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:12:15.942081 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 4dc5794a67374cc388da2edc47cbb113 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:59561)
I20240501 20:12:15.942255 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 4dc5794a67374cc388da2edc47cbb113 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:59561: session id = 565cade0058540a4aa18d7c7a25affbe-4dc5794a67374cc388da2edc47cbb113
I20240501 20:12:15.946012 27154 tablet_copy_source_session.cc:205] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:15.953169 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc5794a67374cc388da2edc47cbb113. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.975569 27504 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:15.981468 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:15.999398 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:16.010964 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.051251 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:16.068711 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:16.070525 2057 tablet_copy_client.cc:783] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:16.070842 2057 tablet_copy_client.cc:647] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:16.093367 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.098116 2057 tablet_copy_client.cc:523] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:16.102602 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:16.127485 2057 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:12:16.141418 27504 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:16.150910 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.159030 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.172544 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:16.190332 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:16.224324 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.232848 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:16.297053 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.318074 27133 consensus_queue.cc:237] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:16.340152 29336 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:12:16.340694 1008 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:16.341624 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:16.349509 27504 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:16.356669 1008 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:16.358625 29335 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:16.363382 26994 catalog_manager.cc:5484] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 2 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:16.400609 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:16.403937 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.459774 2057 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:16.460449 2057 tablet_bootstrap.cc:492] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:12:16.460847 2057 ts_tablet_manager.cc:1367] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.333s user 0.065s sys 0.015s
I20240501 20:12:16.461864 2057 raft_consensus.cc:348] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:16.462255 2057 raft_consensus.cc:729] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:12:16.462600 2057 consensus_queue.cc:260] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:16.464507 2057 ts_tablet_manager.cc:1398] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:16.465788 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-4dc5794a67374cc388da2edc47cbb113 received from {username='slave'} at 127.14.106.197:59561
I20240501 20:12:16.465919 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-4dc5794a67374cc388da2edc47cbb113 on tablet 4dc5794a67374cc388da2edc47cbb113 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:12:16.534667 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:16.544077 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.550848 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:16.582800 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f03e02d13f44d188e5eb2774771a0d with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:16.585160 27133 consensus_queue.cc:237] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:16.588105 29335 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 20:12:16.588924 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7f03e02d13f44d188e5eb2774771a0d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:16.589476 1008 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:16.591780 1008 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:16.596987 29335 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:16.603952 26989 catalog_manager.cc:5484] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc reported cstate change: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 2 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:16.606238 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e7f03e02d13f44d188e5eb2774771a0d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:16.615152 2150 tablet_replica.cc:330] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:16.615381 2150 raft_consensus.cc:2229] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:16.615578 2150 raft_consensus.cc:2258] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:16.616372 2150 ts_tablet_manager.cc:1875] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:16.623849 2150 ts_tablet_manager.cc:1888] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:12:16.628392 2150 log.cc:1196] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/e7f03e02d13f44d188e5eb2774771a0d
I20240501 20:12:16.629329 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet e7f03e02d13f44d188e5eb2774771a0d (table rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]) successfully deleted
I20240501 20:12:16.694254 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.711495 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:16.716809 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:16.724081 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:16.724278 27504 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:12:16.763125 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:16.770298 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:16.779902 29068 raft_consensus.cc:1203] T 4dc5794a67374cc388da2edc47cbb113 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:12:16.809974 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:17.008285 29068 consensus_queue.cc:237] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:17.010926 27395 consensus_queue.cc:237] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:17.014189 27133 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Refusing update from remote peer 565cade0058540a4aa18d7c7a25affbe: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:12:17.016393 1955 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:17.020362 29335 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:17.021075 1942 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:17.023867 27525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74c414c2eed045bda9d85fe5c2fea654"
dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55640
I20240501 20:12:17.024062 27525 raft_consensus.cc:595] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:17.030891 29068 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:17.032121 1942 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:17.035778 27525 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:17.036456 1942 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:17.036140 2231 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:17.037381 27133 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:17.043177 26990 catalog_manager.cc:5484] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 5 to 7, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 2 leader_uuid: "565cade0058540a4aa18d7c7a25affbe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:17.051844 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet 553d443dfb344b2690c260d207901ff1 with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:17.066344 2228 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:17.067590 29069 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:17.072757 29335 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:17.077505 26990 catalog_manager.cc:5484] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:17.084517 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a20d47052f404986315ddb05284ad9 with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:17.091977 2239 tablet_replica.cc:330] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:17.092051 2236 tablet_replica.cc:330] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:17.092259 2236 raft_consensus.cc:2229] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:17.092437 2236 raft_consensus.cc:2258] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:17.092841 2239 raft_consensus.cc:2229] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:17.093448 2239 raft_consensus.cc:2258] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:17.093458 2236 ts_tablet_manager.cc:1875] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:17.099222 2239 ts_tablet_manager.cc:1875] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:17.112079 29069 consensus_queue.cc:237] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:17.115455 27133 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Refusing update from remote peer 565cade0058540a4aa18d7c7a25affbe: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:12:17.116178 1955 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:17.119529 1955 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:17.123711 27133 raft_consensus.cc:2932] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:17.126298 26981 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 553d443dfb344b2690c260d207901ff1 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:17.137163 26989 catalog_manager.cc:5484] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 7 to 8, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 2 leader_uuid: "565cade0058540a4aa18d7c7a25affbe" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:17.147756 2236 ts_tablet_manager.cc:1888] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:12:17.152928 2239 ts_tablet_manager.cc:1888] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:17.153045 2239 log.cc:1196] T b5a20d47052f404986315ddb05284ad9 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/b5a20d47052f404986315ddb05284ad9
I20240501 20:12:17.154762 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet b5a20d47052f404986315ddb05284ad9 (table rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]) successfully deleted
I20240501 20:12:17.155162 2236 log.cc:1196] T 553d443dfb344b2690c260d207901ff1 P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/553d443dfb344b2690c260d207901ff1
W20240501 20:12:17.156739 29003 consensus_peers.cc:485] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 553d443dfb344b2690c260d207901ff1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:17.159847 1420 raft_consensus.cc:979] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: : Instructing follower d8b307156e1346bd9eecf655f49924c7 to start an election
I20240501 20:12:17.160259 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet 553d443dfb344b2690c260d207901ff1 (table rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]) successfully deleted
W20240501 20:12:17.166816 27330 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:43861, user_credentials={real_user=slave}} blocked reactor thread for 59290.1us
W20240501 20:12:17.167492 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 83595.7us
I20240501 20:12:17.233695 2240 ts_tablet_manager.cc:902] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:12:17.235008 2264 raft_consensus.cc:1067] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Signalling peer d8b307156e1346bd9eecf655f49924c7 to start an election
I20240501 20:12:17.239792 27524 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
W20240501 20:12:17.245568 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 74511.8us
W20240501 20:12:17.248463 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 90679.6us
I20240501 20:12:17.250578 27266 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74c414c2eed045bda9d85fe5c2fea654"
dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
from {username='slave'} at 127.14.106.196:35417
I20240501 20:12:17.250880 27266 raft_consensus.cc:482] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:12:17.250988 27266 raft_consensus.cc:3037] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:17.253829 27266 raft_consensus.cc:504] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:17.256965 27266 leader_election.cc:288] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 2 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
I20240501 20:12:17.278340 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c414c2eed045bda9d85fe5c2fea654" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:12:17.278573 27525 raft_consensus.cc:3032] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:12:17.278689 27525 raft_consensus.cc:729] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:12:17.278956 27525 consensus_queue.cc:260] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:17.279414 27525 raft_consensus.cc:3037] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:17.283545 27525 raft_consensus.cc:2454] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 2.
I20240501 20:12:17.286211 29069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c414c2eed045bda9d85fe5c2fea654" candidate_uuid: "d8b307156e1346bd9eecf655f49924c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "565cade0058540a4aa18d7c7a25affbe"
I20240501 20:12:17.286532 29069 raft_consensus.cc:3037] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:17.289579 29069 raft_consensus.cc:2454] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b307156e1346bd9eecf655f49924c7 in term 2.
I20240501 20:12:17.290387 27202 leader_election.cc:302] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 565cade0058540a4aa18d7c7a25affbe, d8b307156e1346bd9eecf655f49924c7; no voters:
I20240501 20:12:17.290782 442 raft_consensus.cc:2781] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:12:17.305159 442 raft_consensus.cc:686] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 2 LEADER]: Becoming Leader. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Running, Role: LEADER
I20240501 20:12:17.305603 442 consensus_queue.cc:237] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:17.380103 26991 catalog_manager.cc:5484] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: term changed from 1 to 2, leader changed from 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) to d8b307156e1346bd9eecf655f49924c7 (127.14.106.194). New cstate: current_term: 2 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:17.383919 2240 tablet_copy_client.cc:308] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:12:17.385761 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet e7f03e02d13f44d188e5eb2774771a0d from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:55267)
I20240501 20:12:17.385952 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet e7f03e02d13f44d188e5eb2774771a0d from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:55267: session id = d56cbcb06b9e418cb617de6a34b946b0-e7f03e02d13f44d188e5eb2774771a0d
I20240501 20:12:17.389160 27154 tablet_copy_source_session.cc:205] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:17.390807 2240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f03e02d13f44d188e5eb2774771a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.399048 2240 tablet_copy_client.cc:783] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:17.399386 2240 tablet_copy_client.cc:647] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:17.402659 27525 consensus_queue.cc:237] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:17.406210 27395 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:17.406534 29203 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:17.407138 2240 tablet_copy_client.cc:523] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:17.407272 2256 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:17.408035 2256 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:17.410838 2240 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:12:17.435535 2256 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:17.440546 27395 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:17.451201 26991 catalog_manager.cc:5484] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:17.456259 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 944d354124664bc0870f251d8a0831c5 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:17.456971 2318 tablet_replica.cc:330] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:17.457229 2318 raft_consensus.cc:2229] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:17.457523 2318 raft_consensus.cc:2258] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:17.458370 2318 ts_tablet_manager.cc:1875] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:17.458360 29203 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:17.501008 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:17.509235 2318 ts_tablet_manager.cc:1888] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:17.509524 2318 log.cc:1196] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/944d354124664bc0870f251d8a0831c5
I20240501 20:12:17.510435 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 944d354124664bc0870f251d8a0831c5 (table rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]) successfully deleted
I20240501 20:12:17.533159 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:17.548754 27504 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:12:17.561005 2240 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:17.562927 2240 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:12:17.564477 2240 ts_tablet_manager.cc:1367] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.154s user 0.072s sys 0.023s
I20240501 20:12:17.565536 2240 raft_consensus.cc:348] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:17.565829 2240 raft_consensus.cc:729] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.577634 2240 consensus_queue.cc:260] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:17.578423 2240 ts_tablet_manager.cc:1398] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:12:17.579339 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-e7f03e02d13f44d188e5eb2774771a0d received from {username='slave'} at 127.14.106.198:55267
I20240501 20:12:17.579471 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-e7f03e02d13f44d188e5eb2774771a0d on tablet e7f03e02d13f44d188e5eb2774771a0d with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:12:17.608274 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:17.620469 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:17.663336 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:17.690307 29069 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:17.691520 442 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:17.702354 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:17.707123 2351 ts_tablet_manager.cc:902] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
I20240501 20:12:17.721364 27525 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:17.722074 442 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:17.794287 2351 tablet_copy_client.cc:308] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.197:40723
I20240501 20:12:17.817418 29090 tablet_copy_service.cc:140] P 565cade0058540a4aa18d7c7a25affbe: Received BeginTabletCopySession request for tablet 553d443dfb344b2690c260d207901ff1 from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:41297)
I20240501 20:12:17.817664 29090 tablet_copy_service.cc:161] P 565cade0058540a4aa18d7c7a25affbe: Beginning new tablet copy session on tablet 553d443dfb344b2690c260d207901ff1 from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:41297: session id = 212de34e3aa24fa1be12c7421df985fc-553d443dfb344b2690c260d207901ff1
I20240501 20:12:17.819614 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:17.821516 29090 tablet_copy_source_session.cc:205] T 553d443dfb344b2690c260d207901ff1 P 565cade0058540a4aa18d7c7a25affbe: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:17.824072 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d443dfb344b2690c260d207901ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.831938 2351 tablet_copy_client.cc:783] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:17.832231 2351 tablet_copy_client.cc:647] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:17.834272 2351 tablet_copy_client.cc:523] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:17.838680 2351 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:12:17.931161 29203 raft_consensus.cc:1203] T e7f03e02d13f44d188e5eb2774771a0d P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:12:18.018427 2351 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:18.019060 2351 tablet_bootstrap.cc:492] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:12:18.019353 2351 ts_tablet_manager.cc:1367] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.181s user 0.049s sys 0.062s
I20240501 20:12:18.023684 2351 raft_consensus.cc:348] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:18.023998 2351 raft_consensus.cc:729] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.024299 2351 consensus_queue.cc:260] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:18.032737 2351 ts_tablet_manager.cc:1398] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.013s user 0.001s sys 0.004s
I20240501 20:12:18.033494 29090 tablet_copy_service.cc:342] P 565cade0058540a4aa18d7c7a25affbe: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-553d443dfb344b2690c260d207901ff1 received from {username='slave'} at 127.14.106.199:41297
I20240501 20:12:18.033675 29090 tablet_copy_service.cc:433] P 565cade0058540a4aa18d7c7a25affbe: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-553d443dfb344b2690c260d207901ff1 on tablet 553d443dfb344b2690c260d207901ff1 with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:12:18.249240 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:18.253741 27504 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:12:18.253849 29335 raft_consensus.cc:1203] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:12:18.277004 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.295464 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:18.307992 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.382186 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:18.399905 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.435317 27525 consensus_queue.cc:237] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:18.439671 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:18.444619 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.453516 27504 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:12:18.454774 29069 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:18.455509 2256 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:18.474759 29465 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:18.475466 2256 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:18.478688 27266 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:18.479776 2265 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:18.485152 2256 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:18.520172 29069 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:18.529518 26991 catalog_manager.cc:5484] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:18.530411 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:18.532212 27266 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:18.535607 29465 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:18.592813 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.594098 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:18.647536 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet af693dc19f844b12973b000093021619 with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:18.659400 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:18.685655 2505 tablet_replica.cc:330] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:18.703640 2505 raft_consensus.cc:2229] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:18.704138 2505 raft_consensus.cc:2258] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:18.706301 2505 ts_tablet_manager.cc:1875] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:18.710453 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:18.727960 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:18.771360 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.772866 27504 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:12:18.788718 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.806196 2505 ts_tablet_manager.cc:1888] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:18.806330 2505 log.cc:1196] T af693dc19f844b12973b000093021619 P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/af693dc19f844b12973b000093021619
I20240501 20:12:18.821717 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.859472 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet af693dc19f844b12973b000093021619 (table rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]) successfully deleted
I20240501 20:12:18.875388 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:18.898952 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.916800 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:18.944275 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:18.951172 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:18.957520 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:18.968251 27504 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:12:18.979637 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:19.027451 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:19.098115 27374 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:19.098753 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:19.136543 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:19.366550 27266 consensus_queue.cc:237] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:19.371459 27525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f"
dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55756
I20240501 20:12:19.371651 27525 raft_consensus.cc:595] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:19.386478 29069 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:12:19.387284 2285 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:19.413386 2265 raft_consensus.cc:979] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: : Instructing follower e677127814314770b3370764d0ff9d06 to start an election
I20240501 20:12:19.413555 2265 raft_consensus.cc:1067] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Signalling peer e677127814314770b3370764d0ff9d06 to start an election
I20240501 20:12:19.414673 27394 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f"
dest_uuid: "e677127814314770b3370764d0ff9d06"
from {username='slave'} at 127.14.106.196:53879
I20240501 20:12:19.414958 27394 raft_consensus.cc:482] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:12:19.415086 27394 raft_consensus.cc:3037] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:19.418956 27394 raft_consensus.cc:504] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:19.420774 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:12:19.421028 27525 raft_consensus.cc:3032] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:12:19.421110 27525 raft_consensus.cc:729] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:12:19.421346 27525 consensus_queue.cc:260] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:19.421761 27525 raft_consensus.cc:3037] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:19.425139 27525 raft_consensus.cc:2454] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 2.
I20240501 20:12:19.425701 2619 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:19.427444 29203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d56cbcb06b9e418cb617de6a34b946b0"
I20240501 20:12:19.427798 29203 raft_consensus.cc:3037] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:19.428849 29069 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:19.427110 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e69dbd245e4b8db397aaffc9145c9f" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:12:19.430088 27266 raft_consensus.cc:3037] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:19.431216 29203 raft_consensus.cc:2454] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 2.
I20240501 20:12:19.432205 27330 leader_election.cc:302] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, d56cbcb06b9e418cb617de6a34b946b0, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:12:19.432507 2261 raft_consensus.cc:2781] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:12:19.432646 2261 raft_consensus.cc:686] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 2 LEADER]: Becoming Leader. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:12:19.432737 27266 raft_consensus.cc:2454] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 2.
I20240501 20:12:19.432948 2261 consensus_queue.cc:237] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:19.434674 26987 catalog_manager.cc:5484] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 2 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:19.436692 27394 leader_election.cc:288] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 2 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843)
I20240501 20:12:19.438930 26985 catalog_manager.cc:5484] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 reported cstate change: term changed from 1 to 2, leader changed from 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) to e677127814314770b3370764d0ff9d06 (127.14.106.195). New cstate: current_term: 2 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:19.455296 27133 consensus_queue.cc:237] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:19.467453 29335 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:19.468079 2343 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:19.473796 29069 raft_consensus.cc:1261] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:19.475255 2343 consensus_queue.cc:1035] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:19.482889 2343 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.487447 29336 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.487776 27133 consensus_queue.cc:237] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:19.524044 26991 catalog_manager.cc:5484] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.524928 29069 raft_consensus.cc:2932] T 3f8902b7b0b649edb10f362713ad36fe P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.551265 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c414c2eed045bda9d85fe5c2fea654 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:19.552515 27266 consensus_queue.cc:237] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:19.556887 29069 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:12:19.557641 2649 tablet_replica.cc:330] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
W20240501 20:12:19.557739 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74c414c2eed045bda9d85fe5c2fea654. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:19.558900 2623 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:19.559027 2649 raft_consensus.cc:2229] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:19.559254 2649 raft_consensus.cc:2258] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:19.560187 2649 ts_tablet_manager.cc:1875] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:19.564021 29336 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:19.564816 2343 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:19.569888 27266 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:19.570672 2343 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:19.575520 29203 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:19.578415 2619 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:19.583480 29069 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:19.588229 2337 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:19.592793 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 74c414c2eed045bda9d85fe5c2fea654 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:19.596915 26994 catalog_manager.cc:5484] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 2 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:19.604353 2337 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.606294 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8902b7b0b649edb10f362713ad36fe with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:19.660737 27266 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.680125 2657 tablet_replica.cc:330] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:19.680505 2657 raft_consensus.cc:2229] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:19.680797 2657 raft_consensus.cc:2258] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:19.681895 2657 ts_tablet_manager.cc:1875] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:19.686158 26994 catalog_manager.cc:5484] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.689918 29336 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.691601 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 5272c34458af43bebb5ba5baefac7137 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:19.695988 29203 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:19.718653 2649 ts_tablet_manager.cc:1888] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:12:19.718802 2649 log.cc:1196] T 74c414c2eed045bda9d85fe5c2fea654 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/74c414c2eed045bda9d85fe5c2fea654
I20240501 20:12:19.719691 2649 tablet_replica.cc:330] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:19.719781 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet 74c414c2eed045bda9d85fe5c2fea654 (table rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]) successfully deleted
I20240501 20:12:19.719936 2649 raft_consensus.cc:2229] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:19.720228 2649 raft_consensus.cc:2258] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:19.721156 2649 ts_tablet_manager.cc:1875] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:19.726910 2657 ts_tablet_manager.cc:1888] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:19.727133 2657 log.cc:1196] T 3f8902b7b0b649edb10f362713ad36fe P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/3f8902b7b0b649edb10f362713ad36fe
I20240501 20:12:19.732771 2649 ts_tablet_manager.cc:1888] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:19.733633 2649 log.cc:1196] T 5272c34458af43bebb5ba5baefac7137 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/5272c34458af43bebb5ba5baefac7137
I20240501 20:12:19.733752 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 3f8902b7b0b649edb10f362713ad36fe (table rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]) successfully deleted
I20240501 20:12:19.734504 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet 5272c34458af43bebb5ba5baefac7137 (table rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]) successfully deleted
I20240501 20:12:19.887085 27266 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:12:19.887876 2394 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:20.011900 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.016474 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:20.031788 29203 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:12:20.032472 2394 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:20.161242 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:20.188439 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:20.204716 2762 ts_tablet_manager.cc:902] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:12:20.270334 27525 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:12:20.270391 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:20.318284 2261 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:20.337071 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.358879 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:20.380292 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.392648 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:20.401408 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.412408 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:20.435600 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.442297 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.449821 2762 tablet_copy_client.cc:308] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:46281
I20240501 20:12:20.450654 27286 tablet_copy_service.cc:140] P d8b307156e1346bd9eecf655f49924c7: Received BeginTabletCopySession request for tablet 74c414c2eed045bda9d85fe5c2fea654 from peer d56cbcb06b9e418cb617de6a34b946b0 ({username='slave'} at 127.14.106.198:55097)
I20240501 20:12:20.450803 27286 tablet_copy_service.cc:161] P d8b307156e1346bd9eecf655f49924c7: Beginning new tablet copy session on tablet 74c414c2eed045bda9d85fe5c2fea654 from peer d56cbcb06b9e418cb617de6a34b946b0 at {username='slave'} at 127.14.106.198:55097: session id = d56cbcb06b9e418cb617de6a34b946b0-74c414c2eed045bda9d85fe5c2fea654
I20240501 20:12:20.453974 27286 tablet_copy_source_session.cc:205] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:20.457376 2762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c414c2eed045bda9d85fe5c2fea654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.482422 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:20.484349 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:20.485150 2762 tablet_copy_client.cc:783] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:20.485457 2762 tablet_copy_client.cc:647] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:20.486121 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:20.505029 2762 tablet_copy_client.cc:523] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:20.519040 2762 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap starting.
I20240501 20:12:20.655797 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.683881 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.684362 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:20.684626 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:20.810370 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:20.841951 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:20.873241 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.915863 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.924285 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:20.951414 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:20.974565 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:20.999819 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:21.012992 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:21.020442 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:21.074419 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:21.129571 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:21.134361 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:21.152159 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:21.165990 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:21.181910 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:21.194317 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:21.249845 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:21.251219 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:21.290477 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:21.305227 2762 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:21.305696 2762 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: Bootstrap complete.
I20240501 20:12:21.306016 2762 ts_tablet_manager.cc:1367] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: Time spent bootstrapping tablet: real 0.797s user 0.053s sys 0.008s
I20240501 20:12:21.310513 27525 consensus_queue.cc:237] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:21.316473 2762 raft_consensus.cc:348] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:21.316695 2762 raft_consensus.cc:729] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56cbcb06b9e418cb617de6a34b946b0, State: Initialized, Role: FOLLOWER
I20240501 20:12:21.316936 2762 consensus_queue.cc:260] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:21.332222 29068 raft_consensus.cc:1261] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:21.332799 2752 consensus_queue.cc:1035] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.347389 2762 ts_tablet_manager.cc:1398] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0: Time spent starting tablet: real 0.041s user 0.004s sys 0.000s
I20240501 20:12:21.349838 27286 tablet_copy_service.cc:342] P d8b307156e1346bd9eecf655f49924c7: Request end of tablet copy session d56cbcb06b9e418cb617de6a34b946b0-74c414c2eed045bda9d85fe5c2fea654 received from {username='slave'} at 127.14.106.198:55097
I20240501 20:12:21.349975 27286 tablet_copy_service.cc:433] P d8b307156e1346bd9eecf655f49924c7: ending tablet copy session d56cbcb06b9e418cb617de6a34b946b0-74c414c2eed045bda9d85fe5c2fea654 on tablet 74c414c2eed045bda9d85fe5c2fea654 with peer d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:12:21.351291 2752 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:21.355368 29069 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:21.360607 26990 catalog_manager.cc:5484] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:21.468806 27394 consensus_queue.cc:237] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:21.493410 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet f02a619bc1584d53ac902c8d8baff976 with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:21.493520 27525 consensus_queue.cc:237] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:21.496400 29068 raft_consensus.cc:1261] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:21.497076 2752 consensus_queue.cc:1035] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:12:21.500453 27459 consensus_peers.cc:485] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f02a619bc1584d53ac902c8d8baff976. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:21.500460 2858 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:21.501191 29068 raft_consensus.cc:2932] T f02a619bc1584d53ac902c8d8baff976 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:21.511013 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f02a619bc1584d53ac902c8d8baff976 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:21.518492 26994 catalog_manager.cc:5484] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 6 to 7, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:21.540257 2942 tablet_replica.cc:330] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:21.540673 2942 raft_consensus.cc:2229] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:21.541097 2942 raft_consensus.cc:2258] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:21.542186 2942 ts_tablet_manager.cc:1875] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:21.553716 29203 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:21.554688 2929 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.566850 27266 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:21.567936 2929 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.576231 2929 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.581183 29203 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.588723 26990 catalog_manager.cc:5484] T 17e69dbd245e4b8db397aaffc9145c9f P d56cbcb06b9e418cb617de6a34b946b0 reported cstate change: config changed from index 4 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 2 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.593921 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 17e69dbd245e4b8db397aaffc9145c9f with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:21.594563 2957 tablet_replica.cc:330] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:21.594837 2957 raft_consensus.cc:2229] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:21.595075 2957 raft_consensus.cc:2258] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:21.595944 2957 ts_tablet_manager.cc:1875] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:21.598802 27266 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.614348 2957 ts_tablet_manager.cc:1888] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:12:21.617149 2942 ts_tablet_manager.cc:1888] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:21.617295 2942 log.cc:1196] T f02a619bc1584d53ac902c8d8baff976 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/f02a619bc1584d53ac902c8d8baff976
I20240501 20:12:21.618014 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet f02a619bc1584d53ac902c8d8baff976 (table rebalance_test_table_0 [id=565e816257734920a19d1af631224041]) successfully deleted
I20240501 20:12:21.628727 2957 log.cc:1196] T 17e69dbd245e4b8db397aaffc9145c9f P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/17e69dbd245e4b8db397aaffc9145c9f
I20240501 20:12:21.630517 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet 17e69dbd245e4b8db397aaffc9145c9f (table rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]) successfully deleted
I20240501 20:12:21.630496 27525 consensus_queue.cc:237] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:21.637254 29203 raft_consensus.cc:1203] T 74c414c2eed045bda9d85fe5c2fea654 P d56cbcb06b9e418cb617de6a34b946b0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:12:21.679209 29465 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:21.680052 2858 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.691763 29203 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:21.692587 2858 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.706022 27266 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:21.707209 2752 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.723016 2752 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.723652 29465 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.728855 26990 catalog_manager.cc:5484] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.730080 27266 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.732932 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet 9721fc9afa4741b78823e293514e6265 with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:21.733424 2942 tablet_replica.cc:330] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:21.733582 2942 raft_consensus.cc:2229] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:21.733806 2942 raft_consensus.cc:2258] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:21.734580 2942 ts_tablet_manager.cc:1875] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:21.735340 29203 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.743543 2942 ts_tablet_manager.cc:1888] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:21.743714 2942 log.cc:1196] T 9721fc9afa4741b78823e293514e6265 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/9721fc9afa4741b78823e293514e6265
I20240501 20:12:21.745060 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet 9721fc9afa4741b78823e293514e6265 (table rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]) successfully deleted
I20240501 20:12:21.811307 27394 consensus_queue.cc:237] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:21.817068 29068 raft_consensus.cc:1261] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:21.817639 27133 raft_consensus.cc:1261] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:21.817788 2929 consensus_queue.cc:1035] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:21.818384 2929 consensus_queue.cc:1035] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:21.826213 2784 raft_consensus.cc:2932] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:21.831593 27133 raft_consensus.cc:2932] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:21.836958 27135 consensus_queue.cc:237] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:21.839843 26994 catalog_manager.cc:5484] T d2160f17e62749dbb41405a895516b06 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:21.847622 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet d2160f17e62749dbb41405a895516b06 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:21.847901 2957 tablet_replica.cc:330] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:21.848064 2957 raft_consensus.cc:2229] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:21.848320 2957 raft_consensus.cc:2258] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:21.849051 2957 ts_tablet_manager.cc:1875] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:21.851800 29069 raft_consensus.cc:2932] T d2160f17e62749dbb41405a895516b06 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:21.865125 2957 ts_tablet_manager.cc:1888] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:21.867208 2957 log.cc:1196] T d2160f17e62749dbb41405a895516b06 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/d2160f17e62749dbb41405a895516b06
I20240501 20:12:21.868438 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet d2160f17e62749dbb41405a895516b06 (table rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]) successfully deleted
I20240501 20:12:21.876215 29203 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:21.876920 2658 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.878409 29335 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:21.880822 2658 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.892578 27266 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:21.893473 2658 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:21.908996 2658 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.954049 29335 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.967083 29203 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P d56cbcb06b9e418cb617de6a34b946b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.973074 26994 catalog_manager.cc:5484] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:21.988126 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 78fcaa70e916442ea9c67cc762c8db6d with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:21.988538 2957 tablet_replica.cc:330] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:21.988724 2957 raft_consensus.cc:2229] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:21.989017 2957 raft_consensus.cc:2258] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:21.989732 2957 ts_tablet_manager.cc:1875] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:22.013274 2957 ts_tablet_manager.cc:1888] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:22.014375 2957 log.cc:1196] T 78fcaa70e916442ea9c67cc762c8db6d P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/78fcaa70e916442ea9c67cc762c8db6d
I20240501 20:12:22.015467 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet 78fcaa70e916442ea9c67cc762c8db6d (table rebalance_test_table_2 [id=92199738005a460797ae03d52d012acf]) successfully deleted
I20240501 20:12:22.028465 3055 ts_tablet_manager.cc:902] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:12:22.031158 27266 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:22.124529 3055 tablet_copy_client.cc:308] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:12:22.125367 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet f02a619bc1584d53ac902c8d8baff976 from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:37233)
I20240501 20:12:22.125540 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet f02a619bc1584d53ac902c8d8baff976 from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:37233: session id = 212de34e3aa24fa1be12c7421df985fc-f02a619bc1584d53ac902c8d8baff976
I20240501 20:12:22.128705 27545 tablet_copy_source_session.cc:205] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:22.130122 3055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02a619bc1584d53ac902c8d8baff976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.150763 3055 tablet_copy_client.cc:783] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:22.151053 3055 tablet_copy_client.cc:647] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:22.160495 3055 tablet_copy_client.cc:523] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:22.166690 3055 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
I20240501 20:12:22.220598 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:22.235025 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:22.266481 3055 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:22.266979 3055 tablet_bootstrap.cc:492] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:12:22.267252 3055 ts_tablet_manager.cc:1367] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.101s user 0.050s sys 0.000s
I20240501 20:12:22.267978 3055 raft_consensus.cc:348] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:22.268173 3055 raft_consensus.cc:729] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.268389 3055 consensus_queue.cc:260] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:22.268960 3055 ts_tablet_manager.cc:1398] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:22.269623 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-f02a619bc1584d53ac902c8d8baff976 received from {username='slave'} at 127.14.106.199:37233
I20240501 20:12:22.269732 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-f02a619bc1584d53ac902c8d8baff976 on tablet f02a619bc1584d53ac902c8d8baff976 with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:12:22.287144 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:22.289988 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:22.300017 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:22.320165 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:22.343267 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:22.353874 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:22.354827 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:22.381691 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:22.399245 29336 raft_consensus.cc:1203] T f02a619bc1584d53ac902c8d8baff976 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:12:22.405210 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:22.434060 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:22.489796 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:22.509667 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:22.581868 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:22.627236 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:22.871084 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:22.872277 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:22.873286 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:22.879949 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:22.885146 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:22.953701 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:22.969628 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:23.047706 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:23.126032 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:23.184629 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:23.195822 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:23.198642 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:23.223731 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:23.229382 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:23.239127 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:23.301745 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:23.536132 27394 consensus_queue.cc:237] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:23.546942 29336 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:23.548013 27525 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:23.548347 2784 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:23.550860 2784 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:23.583968 2784 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:23.589777 29335 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:23.619865 27524 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:23.628377 26985 catalog_manager.cc:5484] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 4 to 5, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:23.689949 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet c619a2235c9248eeb6b2e1d78583d1ba with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:23.690832 3350 tablet_replica.cc:330] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:23.694068 3350 raft_consensus.cc:2229] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:23.694490 3350 raft_consensus.cc:2258] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:23.695499 3350 ts_tablet_manager.cc:1875] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:23.706321 3350 ts_tablet_manager.cc:1888] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:12:23.706554 3350 log.cc:1196] T c619a2235c9248eeb6b2e1d78583d1ba P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/c619a2235c9248eeb6b2e1d78583d1ba
I20240501 20:12:23.707535 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet c619a2235c9248eeb6b2e1d78583d1ba (table rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]) successfully deleted
I20240501 20:12:23.750710 27524 consensus_queue.cc:237] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:23.755026 29335 raft_consensus.cc:1261] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:23.755770 2256 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:23.795861 2256 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:23.799165 29336 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:23.809028 26985 catalog_manager.cc:5484] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 5 to 6, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:23.815214 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet 00e3777032eb4d9c80f47855e5f9f87a with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42090
I20240501 20:12:23.821743 27524 consensus_queue.cc:237] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
W20240501 20:12:23.824503 27460 consensus_peers.cc:485] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00e3777032eb4d9c80f47855e5f9f87a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:23.825161 29336 raft_consensus.cc:1261] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:23.825928 2256 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:23.846207 2256 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:23.852643 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 00e3777032eb4d9c80f47855e5f9f87a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:23.853368 29335 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:23.854488 26985 catalog_manager.cc:5484] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:23.860031 3384 tablet_replica.cc:330] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:12:23.860438 3384 raft_consensus.cc:2229] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:23.860980 3384 raft_consensus.cc:2258] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:23.862452 3384 ts_tablet_manager.cc:1875] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:23.875304 3384 ts_tablet_manager.cc:1888] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:23.875730 3384 log.cc:1196] T 00e3777032eb4d9c80f47855e5f9f87a P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/00e3777032eb4d9c80f47855e5f9f87a
I20240501 20:12:23.878334 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet 00e3777032eb4d9c80f47855e5f9f87a (table rebalance_test_table_8 [id=bd611c8e8104400d8e0a2484f811bbe7]) successfully deleted
I20240501 20:12:23.947003 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:23.960553 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:24.001137 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:24.076408 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:24.082844 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:24.098145 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:24.134788 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:24.136648 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:24.319520 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 44690.5us
I20240501 20:12:24.460856 3520 ts_tablet_manager.cc:902] T 00e3777032eb4d9c80f47855e5f9f87a P d56cbcb06b9e418cb617de6a34b946b0: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
F20240501 20:12:24.461555 3520 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00e3777032eb4d9c80f47855e5f9f87a.queue_time_us, tablet-download-pool-00e3777032eb4d9c80f47855e5f9f87a.queue_time_us) (tablet-download-pool-0c836bb145a54160966c475e1490da38.queue_time_us, tablet-download-pool-0c836bb145a54160966c475e1490da38.queue_time_us) (tablet-download-pool-0c836bb145a54160966c475e1490da38.run_cpu_time_us, tablet-download-pool-0c836bb145a54160966c475e1490da38.run_cpu_time_us) (tablet-download-pool-0c836bb145a54160966c475e1490da38.run_wall_time_us, tablet-download-pool-0c836bb145a54160966c475e1490da38.run_wall_time_us) (tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.queue_time_us, tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.queue_time_us) (tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.run_cpu_time_us, tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.run_cpu_time_us) (tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.run_wall_time_us, tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.run_wall_time_us) (tablet-download-pool-26cb2329d176441ca9127187db2d157a.queue_time_us, tablet-download-pool-26cb2329d176441ca9127187db2d157a.queue_time_us) (tablet-download-pool-26cb2329d176441ca9127187db2d157a.run_cpu_time_us, tablet-download-pool-26cb2329d176441ca9127187db2d157a.run_cpu_time_us) (tablet-download-pool-26cb2329d176441ca9127187db2d157a.run_wall_time_us, tablet-download-pool-26cb2329d176441ca9127187db2d157a.run_wall_time_us) (tablet-download-pool-5272c34458af43bebb5ba5baefac7137.queue_time_us, tablet-download-pool-5272c34458af43bebb5ba5baefac7137.queue_time_us) (tablet-download-pool-5272c34458af43bebb5ba5baefac7137.run_cpu_time_us, tablet-download-pool-5272c34458af43bebb5ba5baefac7137.run_cpu_time_us) (tablet-download-pool-5272c34458af43bebb5ba5baefac7137.run_wall_time_us, tablet-download-pool-5272c34458af43bebb5ba5baefac7137.run_wall_time_us) (tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.queue_time_us, tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.queue_time_us) (tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.run_cpu_time_us, tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.run_cpu_time_us) (tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.run_wall_time_us, tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.run_wall_time_us) (tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.queue_time_us, tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.queue_time_us) (tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.run_cpu_time_us, tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.run_cpu_time_us) (tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.run_wall_time_us, tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.run_wall_time_us) (tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.queue_time_us, tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.queue_time_us) (tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.run_cpu_time_us, tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.run_cpu_time_us) (tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.run_wall_time_us, tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.run_wall_time_us) (tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.queue_time_us, tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.queue_time_us) (tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.run_cpu_time_us, tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.run_cpu_time_us) (tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.run_wall_time_us, tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.run_wall_time_us) (tablet-download-pool-7bb13c1dce8640b28063daa0132d2896.queue_time_us, tablet-download-pool-7bb13c1dce8640b28063daa0132d2896.queue_time_us) (tablet-download-pool-7bb13c1dce8640b28063daa0132d2896.run_cpu_time_us, tablet-download-pool-7bb13c1dce8640b28063daa0132d2896.run_cpu_time_us) (tablet-download-pool-7bb13c1dce8640b28063daa0132d2896.run_wall_time_us, tablet-download-pool-7bb13c1dce8640b28063daa0132d2896.run_wall_time_us) (tablet-download-pool-8bf74c24f28546bb90e10002270e3814.queue_time_us, tablet-download-pool-8bf74c24f28546bb90e10002270e3814.queue_time_us) (tablet-download-pool-8bf74c24f28546bb90e10002270e3814.run_cpu_time_us, tablet-download-pool-8bf74c24f28546bb90e10002270e3814.run_cpu_time_us) (tablet-download-pool-8bf74c24f28546bb90e10002270e3814.run_wall_time_us, tablet-download-pool-8bf74c24f28546bb90e10002270e3814.run_wall_time_us) (tablet-download-pool-944d354124664bc0870f251d8a0831c5.queue_time_us, tablet-download-pool-944d354124664bc0870f251d8a0831c5.queue_time_us) (tablet-download-pool-944d354124664bc0870f251d8a0831c5.run_cpu_time_us, tablet-download-pool-944d354124664bc0870f251d8a0831c5.run_cpu_time_us) (tablet-download-pool-944d354124664bc0870f251d8a0831c5.run_wall_time_us, tablet-download-pool-944d354124664bc0870f251d8a0831c5.run_wall_time_us) (tablet-download-pool-9721fc9afa4741b78823e293514e6265.queue_time_us, tablet-download-pool-9721fc9afa4741b78823e293514e6265.queue_time_us) (tablet-download-pool-9721fc9afa4741b78823e293514e6265.run_cpu_time_us, tablet-download-pool-9721fc9afa4741b78823e293514e6265.run_cpu_time_us) (tablet-download-pool-9721fc9afa4741b78823e293514e6265.run_wall_time_us, tablet-download-pool-9721fc9afa4741b78823e293514e6265.run_wall_time_us) (tablet-download-pool-e5b9f1a087114c82b010bd10e4f676c1.queue_time_us, tablet-download-pool-e5b9f1a087114c82b010bd10e4f676c1.queue_time_us) (tablet-download-pool-e5b9f1a087114c82b010bd10e4f676c1.run_cpu_time_us, tablet-download-pool-e5b9f1a087114c82b010bd10e4f676c1.run_cpu_time_us) (tablet-download-pool-e5b9f1a087114c82b010bd10e4f676c1.run_wall_time_us, tablet-download-pool-e5b9f1a087114c82b010bd10e4f676c1.run_wall_time_us) (tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.queue_time_us, tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.queue_time_us) (tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.run_cpu_time_us, tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.run_cpu_time_us) (tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.run_wall_time_us, tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7ff1dec46dcd google::LogMessage::Fail() at ??:0
@ 0x7ff1dec4ab93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff1dec467cc google::LogMessage::Flush() at ??:0
@ 0x7ff1dec47f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff1e1987742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7ff1e1953f8b kudu::ThreadPool::ThreadPool() at ??:0
W20240501 20:12:24.593585 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 43986.9us
W20240501 20:12:24.594861 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 48296.1us
@ 0x7ff1e194dcd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 20:12:24.610709 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:24.679719 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:24.699048 29183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:24.732288 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7ff1f0da08de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 20:12:24.741523 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:24.777698 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:24.805871 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7ff1f0db8daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 20:12:24.823094 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7ff1f0ee3480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7ff1f0f1ba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7ff1f0efe981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7ff1f0efe70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ff1e1957b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff1e195b19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff1e195b019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 20:12:24.970485 27135 consensus_queue.cc:237] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
@ 0x7ff1e193601e kudu::Thread::SuperviseThread() at ??:0
I20240501 20:12:24.994719 29465 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:24.995563 3484 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7ff1e4cbb6db start_thread at ??:0
I20240501 20:12:25.014827 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7ff1ddaf971f clone at ??:0
I20240501 20:12:25.020351 27394 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:25.020947 3484 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:25.029958 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:25.050772 2899 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:25.052721 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:25.062865 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:25.076797 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:25.101953 29465 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:25.108436 27394 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:25.108076 26989 catalog_manager.cc:5484] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:25.199653 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:25.212683 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf74c24f28546bb90e10002270e3814 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:25.228592 3671 tablet_replica.cc:330] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:25.228811 3671 raft_consensus.cc:2229] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:25.229065 3671 raft_consensus.cc:2258] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:25.229831 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:25.229969 3671 ts_tablet_manager.cc:1875] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:25.290695 3671 ts_tablet_manager.cc:1888] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:25.290832 3671 log.cc:1196] T 8bf74c24f28546bb90e10002270e3814 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/8bf74c24f28546bb90e10002270e3814
I20240501 20:12:25.299880 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 8bf74c24f28546bb90e10002270e3814 (table rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]) successfully deleted
W20240501 20:12:25.555274 27330 connection.cc:535] client connection to 127.14.106.198:46843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:12:25.555545 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:12:25.565637 27460 connection.cc:535] client connection to 127.14.106.198:46843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:12:25.565884 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:12:25.585255 27200 connection.cc:535] client connection to 127.14.106.198:46843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:12:25.585561 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:12:25.586342 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:25.608640 27069 connection.cc:535] client connection to 127.14.106.198:46843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:12:25.608886 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:12:25.626250 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:25.636617 3744 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:12:25.582811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.106.198:46843 (local address 127.14.106.195:46253)
0501 20:12:25.601319 (+ 18508us) negotiation.cc:107] Waiting for socket to connect
0501 20:12:25.601338 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0501 20:12:25.605083 (+ 3745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:12:25.635491 (+ 30408us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":18387,"thread_start_us":18834,"threads_started":1}
W20240501 20:12:25.636818 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.636943 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:25.637715 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:25.649756 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:25.678454 3747 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:12:25.592916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.106.198:46843 (local address 127.14.106.196:50297)
0501 20:12:25.670151 (+ 77235us) negotiation.cc:107] Waiting for socket to connect
0501 20:12:25.670296 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":77041,"thread_start_us":77843,"threads_started":1}
W20240501 20:12:25.678819 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection reset by peer (error 104)
W20240501 20:12:25.678932 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.679008 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.679069 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.679172 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.679248 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.698535 27069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:46843, user_credentials={real_user=slave}} blocked reactor thread for 89704.7us
I20240501 20:12:25.698693 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:25.702172 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:25.718459 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.718613 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.718708 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.718762 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.758821 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.758952 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:25.759035 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:25.797034 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:26.146649 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:26.150334 27524 consensus_queue.cc:237] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:26.185742 29068 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:26.185842 29465 raft_consensus.cc:1261] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:26.186415 3824 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:26.186769 2256 consensus_queue.cc:1035] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:26.231793 3824 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:26.236735 29466 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:26.246704 26985 catalog_manager.cc:5484] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:26.248685 29068 raft_consensus.cc:2932] T af693dc19f844b12973b000093021619 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:26.282088 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.286432 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:26.324748 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet af693dc19f844b12973b000093021619 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:26.332180 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:26.336426 3889 tablet_replica.cc:330] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:26.337705 3889 raft_consensus.cc:2229] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:26.340622 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:26.344287 3889 raft_consensus.cc:2258] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:26.345417 3889 ts_tablet_manager.cc:1875] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:26.373895 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.374372 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:26.377161 27394 consensus_queue.cc:237] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:26.399341 29336 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:26.400029 3535 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:26.403137 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:26.421569 3899 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:26.422760 29336 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:26.431885 26989 catalog_manager.cc:5484] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 5 to 6, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:26.435513 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet c619a2235c9248eeb6b2e1d78583d1ba with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:26.436122 3904 tablet_replica.cc:330] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:26.436599 3904 raft_consensus.cc:2229] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:26.437058 3904 raft_consensus.cc:2258] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:26.438009 3904 ts_tablet_manager.cc:1875] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:26.447551 3904 ts_tablet_manager.cc:1888] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:26.447758 3904 log.cc:1196] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/c619a2235c9248eeb6b2e1d78583d1ba
I20240501 20:12:26.448676 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet c619a2235c9248eeb6b2e1d78583d1ba (table rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]) successfully deleted
I20240501 20:12:26.454819 27394 consensus_queue.cc:237] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:26.472481 3889 ts_tablet_manager.cc:1888] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
W20240501 20:12:26.476617 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:26.478726 29335 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:26.479709 3535 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:26.481235 3889 log.cc:1196] T af693dc19f844b12973b000093021619 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/af693dc19f844b12973b000093021619
I20240501 20:12:26.483851 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet af693dc19f844b12973b000093021619 (table rebalance_test_table_1 [id=fa6385b5bce14289828f8b7da259c90c]) successfully deleted
I20240501 20:12:26.486569 3900 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:26.491247 26981 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c619a2235c9248eeb6b2e1d78583d1ba with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:26.491133 29336 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
I20240501 20:12:26.493789 26989 catalog_manager.cc:5484] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } }
W20240501 20:12:26.597535 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:26.624393 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.645454 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:26.660382 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:26.697474 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:26.710992 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:26.733003 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 74183.4us
I20240501 20:12:26.753440 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.770373 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:26.925316 27524 consensus_queue.cc:237] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:26.939327 29466 raft_consensus.cc:1261] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:26.940028 3824 consensus_queue.cc:1035] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:26.960688 3824 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:26.963766 29466 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:26.974974 26985 catalog_manager.cc:5484] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 5 to 6, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:26.980223 27525 consensus_queue.cc:237] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
W20240501 20:12:26.983844 27459 consensus_peers.cc:485] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0e88937e1ec49bfbaffe535e445e40a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:26.984124 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e88937e1ec49bfbaffe535e445e40a with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54184
I20240501 20:12:26.982686 29466 raft_consensus.cc:1261] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:26.991262 3824 consensus_queue.cc:1035] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:26.993901 2256 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:26.994752 29465 raft_consensus.cc:2932] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:27.003176 4010 tablet_replica.cc:330] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:12:27.003741 4010 raft_consensus.cc:2229] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:27.004043 4010 raft_consensus.cc:2258] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:27.004891 4010 ts_tablet_manager.cc:1875] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:27.010499 26985 catalog_manager.cc:5484] T c0e88937e1ec49bfbaffe535e445e40a P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 6 to 7, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:27.011334 4010 ts_tablet_manager.cc:1888] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:27.011497 4010 log.cc:1196] T c0e88937e1ec49bfbaffe535e445e40a P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/c0e88937e1ec49bfbaffe535e445e40a
I20240501 20:12:27.012638 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet c0e88937e1ec49bfbaffe535e445e40a (table rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]) successfully deleted
I20240501 20:12:27.013437 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c0e88937e1ec49bfbaffe535e445e40a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:27.142165 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:27.175637 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:27.192425 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:27.215880 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:27.220016 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:27.308121 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:27.312738 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:27.352818 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:27.408448 29316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:27.446336 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:27.477116 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:27.480623 27504 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:27.484504 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:27.525128 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:27.531252 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:27.539561 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:27.562712 4109 ts_tablet_manager.cc:902] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:12:27.617367 4109 tablet_copy_client.cc:308] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:12:27.619004 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet c0e88937e1ec49bfbaffe535e445e40a from peer 212de34e3aa24fa1be12c7421df985fc ({username='slave'} at 127.14.106.199:37233)
I20240501 20:12:27.619194 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet c0e88937e1ec49bfbaffe535e445e40a from peer 212de34e3aa24fa1be12c7421df985fc at {username='slave'} at 127.14.106.199:37233: session id = 212de34e3aa24fa1be12c7421df985fc-c0e88937e1ec49bfbaffe535e445e40a
I20240501 20:12:27.622120 27545 tablet_copy_source_session.cc:205] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:27.623750 4109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e88937e1ec49bfbaffe535e445e40a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:12:27.636514 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:27.638046 4109 tablet_copy_client.cc:783] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:27.638593 4109 tablet_copy_client.cc:647] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:27.640722 4109 tablet_copy_client.cc:523] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:27.644541 4109 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: Bootstrap starting.
W20240501 20:12:27.733597 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:27.748389 27524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f034a679870b44caa238f04a8b49d302"
dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37842
I20240501 20:12:27.748651 27524 raft_consensus.cc:595] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Received request to transfer leadership
W20240501 20:12:27.802119 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:27.841962 4109 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:27.842469 4109 tablet_bootstrap.cc:492] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: Bootstrap complete.
I20240501 20:12:27.842783 4109 ts_tablet_manager.cc:1367] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: Time spent bootstrapping tablet: real 0.198s user 0.055s sys 0.028s
I20240501 20:12:27.847894 4109 raft_consensus.cc:348] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:27.848152 4109 raft_consensus.cc:729] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Initialized, Role: FOLLOWER
I20240501 20:12:27.848349 4109 consensus_queue.cc:260] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:27.851668 4109 ts_tablet_manager.cc:1398] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240501 20:12:27.852445 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 212de34e3aa24fa1be12c7421df985fc-c0e88937e1ec49bfbaffe535e445e40a received from {username='slave'} at 127.14.106.199:37233
I20240501 20:12:27.852551 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 212de34e3aa24fa1be12c7421df985fc-c0e88937e1ec49bfbaffe535e445e40a on tablet c0e88937e1ec49bfbaffe535e445e40a with peer 212de34e3aa24fa1be12c7421df985fc
I20240501 20:12:27.879535 4014 raft_consensus.cc:979] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: : Instructing follower 212de34e3aa24fa1be12c7421df985fc to start an election
I20240501 20:12:27.879702 4014 raft_consensus.cc:1067] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Signalling peer 212de34e3aa24fa1be12c7421df985fc to start an election
I20240501 20:12:27.880807 29335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f034a679870b44caa238f04a8b49d302"
dest_uuid: "212de34e3aa24fa1be12c7421df985fc"
from {username='slave'} at 127.14.106.196:54077
I20240501 20:12:27.881346 29335 raft_consensus.cc:482] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:12:27.881531 29335 raft_consensus.cc:3037] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:27.884363 29335 raft_consensus.cc:504] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:27.887149 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f034a679870b44caa238f04a8b49d302" candidate_uuid: "212de34e3aa24fa1be12c7421df985fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:12:27.887377 27135 raft_consensus.cc:3037] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:27.890503 27135 raft_consensus.cc:2454] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212de34e3aa24fa1be12c7421df985fc in term 2.
I20240501 20:12:27.891094 29271 leader_election.cc:302] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212de34e3aa24fa1be12c7421df985fc, 4d603ea5d17441b3bac562bf08ec83fe; no voters:
I20240501 20:12:27.891518 4183 raft_consensus.cc:2781] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:12:27.895402 4183 raft_consensus.cc:686] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 2 LEADER]: Becoming Leader. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Running, Role: LEADER
I20240501 20:12:27.895773 29335 leader_election.cc:288] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [CANDIDATE]: Term 2 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:12:27.895937 4183 consensus_queue.cc:237] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:27.898582 27524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f034a679870b44caa238f04a8b49d302" candidate_uuid: "212de34e3aa24fa1be12c7421df985fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:12:27.898873 27524 raft_consensus.cc:3032] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:12:27.898964 27524 raft_consensus.cc:729] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:12:27.899278 27524 consensus_queue.cc:260] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:27.899694 27524 raft_consensus.cc:3037] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:27.903393 27524 raft_consensus.cc:2454] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212de34e3aa24fa1be12c7421df985fc in term 2.
W20240501 20:12:27.916699 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:27.928216 26990 catalog_manager.cc:5484] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc reported cstate change: term changed from 1 to 2, leader changed from 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) to 212de34e3aa24fa1be12c7421df985fc (127.14.106.199). New cstate: current_term: 2 leader_uuid: "212de34e3aa24fa1be12c7421df985fc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:27.932828 29335 raft_consensus.cc:1203] T c0e88937e1ec49bfbaffe535e445e40a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 20:12:27.953275 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:12:27.964740 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:27.988165 27394 consensus_queue.cc:237] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
W20240501 20:12:28.020834 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:12:28.022550 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:12:28.038486 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:12:28.038705 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:28.039888 29335 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:28.040474 4210 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:28.044029 29068 raft_consensus.cc:1261] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:28.044548 4210 consensus_queue.cc:1035] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:28.045169 4210 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:28.057111 26990 catalog_manager.cc:5484] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:28.065763 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a20d47052f404986315ddb05284ad9 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:28.069875 4258 tablet_replica.cc:330] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:28.070531 4258 raft_consensus.cc:2229] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:28.087815 4258 raft_consensus.cc:2258] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:28.089387 4258 ts_tablet_manager.cc:1875] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:28.091078 29068 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
W20240501 20:12:28.097503 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:12:28.097929 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:28.100519 29335 raft_consensus.cc:2932] T b5a20d47052f404986315ddb05284ad9 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:28.108497 4258 ts_tablet_manager.cc:1888] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:28.111546 4258 log.cc:1196] T b5a20d47052f404986315ddb05284ad9 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/b5a20d47052f404986315ddb05284ad9
I20240501 20:12:28.112421 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet b5a20d47052f404986315ddb05284ad9 (table rebalance_test_table_4 [id=dd89cc8322c34fb1a2bd9dde8dbd37a2]) successfully deleted
W20240501 20:12:28.192385 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:12:28.221810 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:28.315037 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:28.336719 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:28.346694 27525 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Refusing update from remote peer 212de34e3aa24fa1be12c7421df985fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:28.347136 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:28.359565 27135 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Refusing update from remote peer 212de34e3aa24fa1be12c7421df985fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:28.360348 4183 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.365597 4183 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.431382 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:28.516945 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:28.557924 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:28.559092 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:28.645829 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:28.733825 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:28.741137 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:28.745779 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:28.771770 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:28.793526 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:28.824283 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:28.828004 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:28.929586 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:28.930040 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:28.930472 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:12:28.949906 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:28.979395 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:28.987984 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:29.020854 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:29.025643 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:29.083640 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:29.101263 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 57761.1us
W20240501 20:12:29.396242 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:29.786610 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:29.803326 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:29.826804 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:29.895001 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:29.925716 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:29.927659 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:29.944591 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:29.949849 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:29.983379 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:29.987686 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:29.988059 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:12:30.045080 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:30.088534 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:30.131950 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:30.148074 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:30.182389 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:30.210268 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:30.210332 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:30.229926 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:30.264446 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:30.271000 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:30.298132 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:30.320653 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:30.332264 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:30.342855 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:30.355441 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:30.386163 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:30.405179 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:30.411504 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:30.467309 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:30.468838 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:30.479316 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:30.491091 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:30.517371 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:30.544396 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:30.548002 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:30.566512 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:30.583351 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:30.608014 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 20:12:30.625692 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:30.651999 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:12:30.653007 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:12:30.684059 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240501 20:12:30.696738 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 20:12:30.709725 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:12:30.738266 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:12:30.750993 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:12:30.809370 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:12:31.077929 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:31.101596 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:31.102458 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:31.166653 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:31.190374 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:31.277743 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:31.278738 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:31.312247 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:31.457444 29515 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8oH_XT/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 20:11:45.336318 29523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:45.336760 29523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:45.355602 29523 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:48.084431 29523 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:11:49.779440 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:49.786360 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:49.786756 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:49.786890 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:49.786983 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:49.787139 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:49.787245 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:49.787367 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:49.787483 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:49.787645 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:49.787827 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:49.787952 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:49.788061 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:49.788194 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
I20240501 20:11:50.611673 29523 rebalancer_tool.cc:1074] tablet 4dc5794a67374cc388da2edc47cbb113: 'e677127814314770b3370764d0ff9d06' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:50.967403 29523 rebalancer_tool.cc:1074] tablet 309e90bbd9f140a8beca9b3da3f75db4: '524f5e87a3ce40378bde76e667a4ece7' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:51.581127 29523 rebalancer_tool.cc:1074] tablet 26cb2329d176441ca9127187db2d157a: '4d603ea5d17441b3bac562bf08ec83fe' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:52.180409 29523 rebalancer_tool.cc:1074] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'd8b307156e1346bd9eecf655f49924c7' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:52.886713 29523 rebalancer_tool.cc:1074] tablet f034a679870b44caa238f04a8b49d302: 'e677127814314770b3370764d0ff9d06' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:53.379487 29523 rebalancer_tool.cc:1074] tablet 57b4bd50bc8f4d50957bd0280224c420: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:54.025555 29523 rebalancer_tool.cc:1074] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:54.512635 29523 rebalancer_tool.cc:1074] tablet 3f8902b7b0b649edb10f362713ad36fe: 'd8b307156e1346bd9eecf655f49924c7' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:55.204856 29523 rebalancer_tool.cc:1074] tablet 00e3777032eb4d9c80f47855e5f9f87a: 'e677127814314770b3370764d0ff9d06' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:55.648202 29523 rebalancer_tool.cc:1074] tablet 5272c34458af43bebb5ba5baefac7137: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:55.787138 29523 rebalancer_tool.cc:1084] tablet c0e88937e1ec49bfbaffe535e445e40a: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move ignored: Illegal state: tablet c0e88937e1ec49bfbaffe535e445e40a: replica e957d29835544ac587917cc067fb7423 is already present
I20240501 20:11:55.787304 29523 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:57.695439 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:57.695816 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:57.695992 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:57.696177 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:57.696280 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:57.696395 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:57.696592 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:57.696732 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:57.696930 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:57.697063 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:57.697168 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:57.852376 29523 rebalancer_tool.cc:1074] tablet b5a20d47052f404986315ddb05284ad9: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:11:57.998096 29523 rebalancer_tool.cc:1074] tablet af693dc19f844b12973b000093021619: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:11:58.496482 29523 rebalancer_tool.cc:1074] tablet ebcdab26b4ed45f596282d4805d1bb1f: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:58.920241 29523 rebalancer_tool.cc:1074] tablet d2160f17e62749dbb41405a895516b06: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:59.043732 29523 rebalancer_tool.cc:1074] tablet f02a619bc1584d53ac902c8d8baff976: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:11:59.502992 29523 rebalancer_tool.cc:1074] tablet 944d354124664bc0870f251d8a0831c5: 'd8b307156e1346bd9eecf655f49924c7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:12:00.041814 29523 rebalancer_tool.cc:1074] tablet 0c836bb145a54160966c475e1490da38: 'd8b307156e1346bd9eecf655f49924c7' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:12:00.319345 29523 rebalancer_tool.cc:1074] tablet 74c414c2eed045bda9d85fe5c2fea654: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:00.892479 29523 rebalancer_tool.cc:1084] tablet 17e69dbd245e4b8db397aaffc9145c9f: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 20:12:00.892647 29523 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:12:02.938643 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:12:02.938843 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:02.939070 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:02.939200 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:12:02.939332 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:02.939514 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:12:02.939710 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:02.939816 29523 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
I20240501 20:12:03.097074 29523 rebalancer_tool.cc:1074] tablet 7bb13c1dce8640b28063daa0132d2896: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:11.352615 29523 rebalancer_tool.cc:1136] tablet 4dc5794a67374cc388da2edc47cbb113: 'e677127814314770b3370764d0ff9d06' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:15.408933 29523 rebalancer_tool.cc:1136] tablet 74c414c2eed045bda9d85fe5c2fea654: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:19.723753 29523 rebalancer_tool.cc:1136] tablet 57b4bd50bc8f4d50957bd0280224c420: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:23.889674 29523 rebalancer_tool.cc:1136] tablet 5272c34458af43bebb5ba5baefac7137: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:24.069499 29523 rebalancer_tool.cc:1136] tablet b5a20d47052f404986315ddb05284ad9: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
W20240501 20:12:25.556828 3615 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:12:25.011301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.106.198:46843 (local address 127.0.0.1:55300)
0501 20:12:25.013089 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0501 20:12:25.013104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:12:25.013181 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:12:25.163140 (+149959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:12:25.163384 (+ 244us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:12:25.163419 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:12:25.163871 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:12:25.163887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:12:25.555122 (+391235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1715}
W20240501 20:12:25.616374 3615 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:12:25.559932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.106.198:46843 (local address 127.0.0.1:55306)
0501 20:12:25.562092 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0501 20:12:25.562108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 20:12:25.562209 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:12:25.615894 (+ 53685us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2051}
I20240501 20:12:26.287178 29523 rebalancer_tool.cc:1136] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'd8b307156e1346bd9eecf655f49924c7' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:26.477186 29523 rebalancer_tool.cc:1136] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move completed: OK
I20240501 20:12:26.748334 29523 rebalancer_tool.cc:1136] tablet ebcdab26b4ed45f596282d4805d1bb1f: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move completed: OK
I20240501 20:12:26.823611 29523 rebalancer_tool.cc:1074] tablet c0e88937e1ec49bfbaffe535e445e40a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:26.872792 29523 rebalancer_tool.cc:1074] tablet e7f03e02d13f44d188e5eb2774771a0d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:27.078441 29523 rebalancer_tool.cc:1136] tablet e7f03e02d13f44d188e5eb2774771a0d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:27.194201 29523 rebalancer_tool.cc:1136] tablet c0e88937e1ec49bfbaffe535e445e40a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:27.265309 29523 rebalancer_tool.cc:1136] tablet 0c836bb145a54160966c475e1490da38: 'd8b307156e1346bd9eecf655f49924c7' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:27.608768 29523 rebalancer_tool.cc:1136] tablet f034a679870b44caa238f04a8b49d302: 'e677127814314770b3370764d0ff9d06' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:27.686506 29523 rebalancer_tool.cc:1136] tablet f02a619bc1584d53ac902c8d8baff976: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:27.788846 29523 rebalancer_tool.cc:1136] tablet 26cb2329d176441ca9127187db2d157a: '4d603ea5d17441b3bac562bf08ec83fe' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:27.846468 29523 rebalancer_tool.cc:1136] tablet 7bb13c1dce8640b28063daa0132d2896: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:27.908646 29523 rebalancer_tool.cc:1136] tablet 309e90bbd9f140a8beca9b3da3f75db4: '524f5e87a3ce40378bde76e667a4ece7' -> 'e957d29835544ac587917cc067fb7423' move completed: OK
I20240501 20:12:27.999352 29523 rebalancer_tool.cc:1136] tablet 944d354124664bc0870f251d8a0831c5: 'd8b307156e1346bd9eecf655f49924c7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:28.066705 29523 rebalancer_tool.cc:1136] tablet 3f8902b7b0b649edb10f362713ad36fe: 'd8b307156e1346bd9eecf655f49924c7' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:28.124953 29523 rebalancer_tool.cc:1136] tablet d2160f17e62749dbb41405a895516b06: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:28.296761 29523 rebalancer_tool.cc:1136] tablet 00e3777032eb4d9c80f47855e5f9f87a: 'e677127814314770b3370764d0ff9d06' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:28.382251 29523 rebalancer_tool.cc:1136] tablet af693dc19f844b12973b000093021619: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:28.515913 29523 rebalancer_tool.cc:1074] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:28.685251 29523 rebalancer_tool.cc:1074] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:28.957293 29523 rebalancer_tool.cc:1074] tablet c619a2235c9248eeb6b2e1d78583d1ba: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:29.104519 29523 rebalancer_tool.cc:1136] tablet c619a2235c9248eeb6b2e1d78583d1ba: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:29.239732 29523 rebalancer_tool.cc:1136] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:29.322031 29523 rebalancer_tool.cc:1136] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
Illegal state: tablet server d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): unacceptable health status UNAVAILABLE
I20240501 20:12:31.490867 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:31.512763 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:31.519774 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:31.520512 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:31.524662 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:31.533236 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:12:31.541637 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:31.552851 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:31.580041 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:31.745576 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:31.766163 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:31.808439 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:31.812374 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:31.826388 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:31.877645 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:31.878247 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:31.893113 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:31.913023 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:31.913069 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:31.918900 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:31.925132 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:31.970532 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:32.054982 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:32.084705 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:32.195809 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:32.203822 29336 consensus_queue.cc:237] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:32.225879 27135 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Refusing update from remote peer 212de34e3aa24fa1be12c7421df985fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:12:32.226728 4352 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.240533 27135 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:32.248358 26994 catalog_manager.cc:5484] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 2 leader_uuid: "212de34e3aa24fa1be12c7421df985fc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:32.239838 4967 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:12:32.304335 29336 consensus_queue.cc:237] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:32.307250 27135 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Refusing update from remote peer 212de34e3aa24fa1be12c7421df985fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 20:12:32.308123 29272 consensus_peers.cc:485] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f034a679870b44caa238f04a8b49d302. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:32.308317 4967 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:32.310768 4352 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:32.311295 27135 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:32.316150 26978 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f034a679870b44caa238f04a8b49d302 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:32.317672 26990 catalog_manager.cc:5484] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 7 to 8, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 2 leader_uuid: "212de34e3aa24fa1be12c7421df985fc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:32.325556 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet f034a679870b44caa238f04a8b49d302 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47514
I20240501 20:12:32.329146 4989 tablet_replica.cc:330] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:12:32.329341 4989 raft_consensus.cc:2229] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:32.329512 4989 raft_consensus.cc:2258] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:32.331389 4989 ts_tablet_manager.cc:1875] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:32.339537 4989 ts_tablet_manager.cc:1888] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:12:32.339727 4989 log.cc:1196] T f034a679870b44caa238f04a8b49d302 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/f034a679870b44caa238f04a8b49d302
I20240501 20:12:32.340672 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet f034a679870b44caa238f04a8b49d302 (table rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]) successfully deleted
W20240501 20:12:32.497846 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:32.535090 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:12:32.595335 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:32.625118 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:32.628475 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:32.647168 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:32.674137 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:32.682303 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:32.716369 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:32.785647 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:32.820780 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:32.826517 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:32.841679 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:32.846179 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:32.868480 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:32.888959 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:32.896009 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:32.898113 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:32.910465 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:32.919548 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:32.946449 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:32.964064 5096 ts_tablet_manager.cc:902] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
I20240501 20:12:32.964329 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:33.041740 5096 tablet_copy_client.cc:308] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.199:32969
I20240501 20:12:33.043305 29356 tablet_copy_service.cc:140] P 212de34e3aa24fa1be12c7421df985fc: Received BeginTabletCopySession request for tablet f034a679870b44caa238f04a8b49d302 from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:54965)
I20240501 20:12:33.043556 29356 tablet_copy_service.cc:161] P 212de34e3aa24fa1be12c7421df985fc: Beginning new tablet copy session on tablet f034a679870b44caa238f04a8b49d302 from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:54965: session id = 565cade0058540a4aa18d7c7a25affbe-f034a679870b44caa238f04a8b49d302
I20240501 20:12:33.050215 29356 tablet_copy_source_session.cc:205] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:33.060519 5096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f034a679870b44caa238f04a8b49d302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:33.060730 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:33.095816 5096 tablet_copy_client.cc:783] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:33.096196 5096 tablet_copy_client.cc:647] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:33.100389 5096 tablet_copy_client.cc:523] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:33.106281 5096 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
W20240501 20:12:33.132539 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:12:33.137468 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:12:33.203068 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:33.252446 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:33.253934 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:33.259691 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:33.280752 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:33.307650 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:33.313555 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:33.326872 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:33.350697 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:33.360517 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:33.370805 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:33.377140 5096 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:33.377568 5096 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:12:33.382532 5096 ts_tablet_manager.cc:1367] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.278s user 0.050s sys 0.008s
I20240501 20:12:33.383782 5096 raft_consensus.cc:348] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:33.383982 5096 raft_consensus.cc:729] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:12:33.386524 5096 consensus_queue.cc:260] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:12:33.389282 5096 ts_tablet_manager.cc:1398] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20240501 20:12:33.392648 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:33.393368 29356 tablet_copy_service.cc:342] P 212de34e3aa24fa1be12c7421df985fc: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-f034a679870b44caa238f04a8b49d302 received from {username='slave'} at 127.14.106.197:54965
I20240501 20:12:33.393491 29356 tablet_copy_service.cc:433] P 212de34e3aa24fa1be12c7421df985fc: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-f034a679870b44caa238f04a8b49d302 on tablet f034a679870b44caa238f04a8b49d302 with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:12:33.402184 29069 raft_consensus.cc:1203] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:12:33.772061 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:33.795356 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:33.798156 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:33.803558 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:33.824185 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:33.838989 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:33.876444 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:33.886965 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:33.901489 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:33.908815 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:33.925015 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:33.965716 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:33.976754 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:12:34.005124 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:34.007033 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:34.026425 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:34.202809 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:34.303238 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 61914.8us
I20240501 20:12:34.403002 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:34.409555 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:34.431489 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:34.441187 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:34.447562 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:34.464041 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 54767.5us
I20240501 20:12:34.464253 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:34.511013 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:34.582954 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:34.596200 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:34.621722 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:34.632274 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:34.661461 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:34.686805 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:34.720583 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:34.735669 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:34.796933 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:34.811096 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:34.930413 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:12:35.003144 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:35.020010 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:35.024037 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:35.074723 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:35.086356 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:35.101998 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:35.118870 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:35.119400 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:35.149638 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:35.180538 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:12:35.188097 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:35.214689 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:35.227602 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:35.234766 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:35.237844 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:35.274951 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:35.297585 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:35.302065 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:35.313198 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:12:35.421260 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:35.428916 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:35.432468 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:35.437780 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:12:35.438490 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:35.471791 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:35.475541 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:35.475678 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:35.558928 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:35.604593 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:35.634204 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:35.652181 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:12:35.654124 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:35.689450 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:12:35.724483 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:12:35.725692 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 48370.6us
I20240501 20:12:35.729012 29336 consensus_queue.cc:237] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
W20240501 20:12:35.744587 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:12:35.771579 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 20:12:35.772580 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:12:35.782369 29069 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Refusing update from remote peer 212de34e3aa24fa1be12c7421df985fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:12:35.783193 4967 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:35.784155 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:35.801640 5492 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:35.802456 29069 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:35.808470 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:35.827162 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:35.831091 26989 catalog_manager.cc:5484] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc reported cstate change: config changed from index 8 to 9, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 2 leader_uuid: "212de34e3aa24fa1be12c7421df985fc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:12:35.835495 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:35.845630 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:35.874807 27114 tablet_service.cc:1514] Processing DeleteTablet for tablet f034a679870b44caa238f04a8b49d302 with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51516
I20240501 20:12:35.877365 29336 consensus_queue.cc:237] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:35.882229 29069 raft_consensus.cc:1261] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Refusing update from remote peer 212de34e3aa24fa1be12c7421df985fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:12:35.883950 4967 consensus_queue.cc:1035] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:35.893268 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:35.897696 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:35.899145 4967 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:35.907696 29069 raft_consensus.cc:2932] T f034a679870b44caa238f04a8b49d302 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:35.925607 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:35.948094 26978 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f034a679870b44caa238f04a8b49d302 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:12:35.953253 26990 catalog_manager.cc:5484] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc reported cstate change: config changed from index 9 to 10, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 2 leader_uuid: "212de34e3aa24fa1be12c7421df985fc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:12:36.014176 5504 tablet_replica.cc:330] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:12:36.014600 5504 raft_consensus.cc:2229] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:36.015024 5504 raft_consensus.cc:2258] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:36.016207 5504 ts_tablet_manager.cc:1875] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:12:36.062762 29271 consensus_peers.cc:485] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f034a679870b44caa238f04a8b49d302. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:36.071985 5504 ts_tablet_manager.cc:1888] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 20:12:36.072238 5504 log.cc:1196] T f034a679870b44caa238f04a8b49d302 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/f034a679870b44caa238f04a8b49d302
I20240501 20:12:36.073518 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet f034a679870b44caa238f04a8b49d302 (table rebalance_test_table_3 [id=8e80affb3a7b40bda2dfa097a6c40afa]) successfully deleted
I20240501 20:12:36.269834 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:36.287359 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:36.300136 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:36.308212 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:36.378974 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:36.386816 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:36.386930 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:36.404327 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:36.404999 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:36.418239 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:36.431635 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:36.450665 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:36.459172 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:36.459517 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:12:36.465948 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:36.484122 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:36.493933 5585 ts_tablet_manager.cc:902] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
I20240501 20:12:36.532804 5585 tablet_copy_client.cc:308] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.199:32969
I20240501 20:12:36.564952 29356 tablet_copy_service.cc:140] P 212de34e3aa24fa1be12c7421df985fc: Received BeginTabletCopySession request for tablet f034a679870b44caa238f04a8b49d302 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:50013)
I20240501 20:12:36.565161 29356 tablet_copy_service.cc:161] P 212de34e3aa24fa1be12c7421df985fc: Beginning new tablet copy session on tablet f034a679870b44caa238f04a8b49d302 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:50013: session id = e957d29835544ac587917cc067fb7423-f034a679870b44caa238f04a8b49d302
I20240501 20:12:36.568110 29356 tablet_copy_source_session.cc:205] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:36.574762 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f034a679870b44caa238f04a8b49d302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:36.581476 5585 tablet_copy_client.cc:783] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:36.581808 5585 tablet_copy_client.cc:647] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:36.583531 5585 tablet_copy_client.cc:523] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:36.592511 5585 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:12:36.694200 5585 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:36.694675 5585 tablet_bootstrap.cc:492] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:12:36.694975 5585 ts_tablet_manager.cc:1367] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.103s user 0.052s sys 0.002s
I20240501 20:12:36.695642 5585 raft_consensus.cc:348] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:36.695817 5585 raft_consensus.cc:729] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:12:36.696033 5585 consensus_queue.cc:260] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:36.700057 5585 ts_tablet_manager.cc:1398] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:12:36.704114 29356 tablet_copy_service.cc:342] P 212de34e3aa24fa1be12c7421df985fc: Request end of tablet copy session e957d29835544ac587917cc067fb7423-f034a679870b44caa238f04a8b49d302 received from {username='slave'} at 127.14.106.200:50013
I20240501 20:12:36.704254 29356 tablet_copy_service.cc:433] P 212de34e3aa24fa1be12c7421df985fc: ending tablet copy session e957d29835544ac587917cc067fb7423-f034a679870b44caa238f04a8b49d302 on tablet f034a679870b44caa238f04a8b49d302 with peer e957d29835544ac587917cc067fb7423
W20240501 20:12:36.848232 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:37.008574 29465 raft_consensus.cc:1203] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 20:12:37.230788 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:37.239409 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:12:37.261997 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:37.264775 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:37.266484 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:37.272224 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:37.300925 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:37.333983 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:37.339426 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:37.357159 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:37.362536 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:37.365679 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:37.375595 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:37.382004 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.387777 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:37.403880 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:37.451471 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:37.481184 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:37.504426 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.557255 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:37.573832 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:37.584582 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:37.599927 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:37.607761 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:37.619423 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:37.630661 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:37.638489 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:37.642174 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:37.662153 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:37.688973 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:37.697675 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.720209 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:37.724454 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:37.735383 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:12:37.735663 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:12:37.741272 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:37.762563 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:37.965266 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:12:38.139447 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:38.156831 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:38.191150 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:38.191412 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 59149.8us
W20240501 20:12:38.210011 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:12:38.223507 27068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 144222us
I20240501 20:12:38.238723 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:38.294186 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:38.328059 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:38.341797 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:38.344795 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:38.386341 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:38.391202 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:38.402109 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:12:38.407871 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:38.413162 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:38.419304 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:38.420496 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:38.426967 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:38.465318 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:38.486055 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:38.490921 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:38.591742 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:38.609532 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:38.621474 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:38.665056 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:38.672175 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:38.701568 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:38.710898 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:38.715930 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:38.776815 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:38.873363 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:38.882925 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:38.893688 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:38.908358 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:38.967001 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:38.967965 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:38.979282 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:12:38.986191 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:39.181996 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:39.191465 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:39.207625 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:39.248708 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:39.253605 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:39.306965 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:39.310138 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:39.343580 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:39.514662 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:39.578563 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:39.665553 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:39.679302 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:39.697168 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:39.698895 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:39.711179 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:39.711323 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:39.768847 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:39.795383 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:39.812826 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:39.817435 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:39.822486 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:39.824522 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:39.858826 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:39.864233 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:39.885797 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:39.893074 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:39.915581 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:12:39.917831 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:39.935113 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:39.940287 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:39.966872 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:12:39.972748 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:39.981338 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:40.083668 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:40.113626 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:40.167379 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:40.176051 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:40.245697 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:12:40.290928 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:40.310340 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:40.313936 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:40.317456 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:40.325263 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:40.349525 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:40.367221 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:40.536460 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:40.557240 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:40.584955 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:40.606542 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:40.734881 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:12:40.805125 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:12:40.808145 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:40.816479 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:40.819255 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:40.825135 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:40.833295 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:12:40.833644 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:12:40.834988 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:40.835215 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:12:40.849619 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:40.853964 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:40.873911 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:40.898998 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:40.908660 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:40.927053 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:40.953125 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:40.957372 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:40.959961 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:41.013775 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:41.030566 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:12:41.050384 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:41.072923 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:12:41.187870 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 95545.8us
I20240501 20:12:41.215149 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:41.224388 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:41.235843 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:12:41.257907 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:41.268150 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:41.288563 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:41.312428 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:41.316253 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:41.416249 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:41.425036 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:41.435295 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:41.453909 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:41.488914 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:41.490159 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:41.533672 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:41.557662 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:41.796873 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:41.860675 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:41.876259 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:41.929682 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:41.942286 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:41.971442 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:41.982354 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:41.994930 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:42.107692 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:42.142313 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:42.146878 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:42.173928 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:42.179953 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:42.185753 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:42.210827 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:12:42.262430 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:42.329867 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:12:42.372084 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:42.382074 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:42.385085 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:12:42.405480 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:42.418339 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:42.428267 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:42.460613 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:42.466110 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:12:42.466234 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:12:42.466845 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:42.483546 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:12:42.488407 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:42.579452 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:42.606401 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:42.624668 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:42.653761 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:42.666427 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:42.684906 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:42.707516 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:42.723506 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:12:42.737586 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:12:42.782686 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:42.852219 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:42.860460 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:42.873500 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:42.882858 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:42.891499 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:42.902264 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:12:42.919775 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:42.955122 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:42.956293 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:12:43.000264 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:12:43.259603 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:12:43.268775 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:43.469169 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:43.470007 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:12:43.471607 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:43.493744 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:43.518038 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:43.571553 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:43.592085 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:43.626844 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:43.658957 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:43.684981 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:43.685109 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:43.731148 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:43.766991 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:43.782176 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 118471us
I20240501 20:12:43.786808 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:43.797582 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:43.801302 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:12:43.803560 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:43.833813 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:43.852062 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:43.853690 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:43.900050 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:43.924710 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:43.925241 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:43.946779 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:43.997264 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:12:44.053123 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:44.151752 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:44.192899 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:44.197546 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:44.230819 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:44.234732 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:44.250449 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:44.275728 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:44.467150 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:44.496196 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:44.506204 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:44.520345 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:44.545634 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:44.578465 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:44.636262 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:44.643754 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 66057.6us
I20240501 20:12:44.652631 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:44.723563 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:12:44.743439 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:12:44.759658 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:44.790585 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:44.813997 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:44.826494 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:44.841558 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:44.854252 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:44.874601 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:44.892307 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:44.895524 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:44.945300 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:12:44.946007 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:45.023298 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:45.040220 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:45.045881 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:45.055585 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:45.070324 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:45.089555 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:45.093879 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:45.124009 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:45.194535 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:45.211584 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:45.243057 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:45.378876 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:45.415215 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:45.419808 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:45.428100 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:45.450873 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:45.464219 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:45.474885 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:45.478188 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:45.485093 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:45.672282 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:12:45.741875 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:45.769441 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:45.789602 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:45.800776 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:45.801163 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:45.851346 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:45.855367 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 20:12:45.861380 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:45.917271 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:45.922796 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:45.952507 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240501 20:12:45.953797 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:46.004578 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:12:46.059538 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:12:46.086732 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.089222 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:46.100253 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:46.114118 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.165566 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:46.217547 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:46.242491 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:46.250941 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.264686 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.265702 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:46.284643 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:46.286149 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:46.287007 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:46.295148 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:46.340252 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:46.508940 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:46.553161 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.577639 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.591748 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:46.598390 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:46.599200 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:46.612730 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:46.651201 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:12:46.663525 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:46.816749 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.833642 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:46.855072 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:46.875262 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.893471 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:46.946123 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:46.967213 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:47.068310 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:47.129530 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:47.146193 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:47.151944 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:47.174014 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:47.198716 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:47.224385 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:47.252902 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:47.266371 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:47.287726 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:47.293118 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:47.299474 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:47.304214 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:47.331316 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:47.343209 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:47.391517 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:47.414062 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:47.419358 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:12:47.433053 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:47.440166 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:47.514247 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:12:47.616151 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:47.636361 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:47.708798 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:47.734807 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:47.750087 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:47.785953 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:47.804242 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:47.817217 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:47.818260 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:47.822904 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:47.836202 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:47.846843 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:47.868404 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:47.875337 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:47.893038 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:47.905444 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:47.952105 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:48.103492 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:48.112370 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:48.179509 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:48.211751 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:48.218827 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:48.241251 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:48.245739 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:48.257180 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:48.282585 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:48.309537 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:48.347081 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:12:48.460327 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:48.517109 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:48.546873 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:48.554442 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:48.579512 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:12:48.585816 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:48.596940 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:48.606386 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:48.622879 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:48.631695 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:48.642338 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:48.874444 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:48.918715 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:12:48.921665 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:48.923928 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:48.929237 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:48.941555 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:48.994938 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:49.095741 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:49.100243 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:49.161381 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:49.177697 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:49.195261 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:49.233371 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:49.239379 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:49.240150 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:49.254544 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:49.466478 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:49.472079 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:49.497915 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:49.503334 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:49.545789 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:49.548409 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:49.559774 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:49.609603 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 50002.6us
W20240501 20:12:49.646448 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:12:49.712507 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:49.760511 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:49.765244 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:49.812736 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:49.838291 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:49.866725 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:49.868351 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:49.870154 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:49.881039 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:49.924203 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:49.961179 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:50.051921 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:12:50.071105 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:12:50.077903 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:12:50.139346 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:50.186897 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:50.192571 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:50.196537 29272 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 67075.8us
I20240501 20:12:50.220603 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:50.221284 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:50.246840 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:50.247334 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:50.280393 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:50.316220 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:50.387490 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:50.399609 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:50.399634 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:50.423133 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 48968.2us
I20240501 20:12:50.443280 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:50.462345 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:50.478794 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:50.484761 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:50.485128 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:50.489282 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:50.508425 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:50.573320 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:50.581949 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:50.588163 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:50.620565 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:50.638895 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:50.709617 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:50.756186 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:50.873721 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:50.897500 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 20:12:50.907732 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:50.919551 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:50.929164 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:50.942752 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:50.949517 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:50.975436 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:51.009564 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:51.027274 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:51.036378 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:51.049073 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:12:51.133440 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:12:51.191464 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:12:51.199169 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:12:51.206497 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:51.293795 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:51.314962 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:51.347101 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:51.367431 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:51.400941 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:51.419600 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:51.423655 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:51.621608 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:12:51.677253 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:51.687155 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:51.709229 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:51.744073 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:51.747895 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:51.780100 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:51.826959 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:51.827944 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:51.852166 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:51.878937 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:51.903281 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:51.903563 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:51.910519 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:51.940310 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:51.966871 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:51.978663 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:12:52.020896 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:12:52.186697 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:52.222779 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:52.232806 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:52.237820 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:52.258575 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:52.261758 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:52.278774 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:52.297672 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:52.361219 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:52.362409 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:52.432862 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:52.453377 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:52.468616 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:52.502501 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:52.539222 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:52.556448 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:52.571519 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:52.598351 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:52.600685 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:52.647328 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:52.716941 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:52.730695 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:52.744431 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:52.769987 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:52.776520 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:12:52.777377 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:52.808454 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:52.809486 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:52.878396 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:52.998402 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:53.002818 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:53.010214 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:53.036147 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:53.098773 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:53.102213 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:53.126840 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:53.143718 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:12:53.238428 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:12:53.278385 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:53.346782 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:53.373103 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:53.375558 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:53.412175 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:53.412292 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:53.443652 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:53.451371 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:53.482241 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:12:53.544106 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:53.613354 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:53.648002 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:53.652084 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:53.665266 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:53.694039 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:53.697228 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:53.712548 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:53.724372 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:53.744947 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:53.793830 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:53.954463 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:53.967463 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:54.013495 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:54.017326 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:54.045444 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:54.047353 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:54.110316 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:12:54.110836 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:54.272934 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:54.320616 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:54.377895 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:54.380892 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:54.420823 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:54.459733 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:54.509439 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:12:54.532706 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:54.537606 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:54.552628 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:54.651867 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:54.663507 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:54.691181 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:54.719959 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:54.728020 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:54.766734 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:54.775445 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:54.846415 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:54.846475 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:54.847757 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:54.859055 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:54.886655 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:54.888355 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:54.919153 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:12:54.940956 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:54.960625 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:54.963819 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:55.029562 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:55.048408 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:55.053714 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:55.116117 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:55.143711 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:12:55.146878 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:55.151849 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:55.158954 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:55.292117 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:55.304674 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:55.365306 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:55.401263 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:55.524011 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:55.526275 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:55.528743 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:55.559000 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:55.590088 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:55.594399 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:55.693882 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:55.724540 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:55.735332 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 50120.7us
I20240501 20:12:55.747392 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:55.760039 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:55.787714 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:55.790855 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:55.815920 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:12:55.816327 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:55.865584 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:55.868773 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:55.870283 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:55.880918 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:55.899178 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:55.902424 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:55.931494 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:55.934798 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:55.935693 29272 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 79826.7us
I20240501 20:12:55.940254 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:55.983312 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:56.005944 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:56.045434 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:56.054702 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:12:56.059140 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:56.087350 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:12:56.088286 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:56.253624 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:12:56.317785 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:56.323889 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:12:56.324888 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:56.366677 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:56.380915 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:56.407143 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:56.468377 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:56.512727 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:56.520104 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:56.526602 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:12:56.738824 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:56.984172 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:56.987560 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:57.017992 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:57.106666 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:57.137624 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:57.145035 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:57.148626 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:57.165186 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:57.214638 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:57.216740 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:57.237659 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:57.243392 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:57.250977 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:57.264050 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:57.267845 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:57.280539 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:57.282941 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:57.345541 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:57.350010 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:57.350811 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:57.358863 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:57.368186 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:57.374042 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:57.399914 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:57.420814 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:57.482079 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:57.659448 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:57.668576 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:57.677325 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:57.682225 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:57.701994 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:57.727648 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:57.764173 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:57.770677 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:57.953653 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:57.979257 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:57.999842 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:58.006958 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:58.020536 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:58.032219 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:58.034408 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:12:58.036442 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:58.060114 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:58.088445 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:58.135739 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:58.237502 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:12:58.254947 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:12:58.319550 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:58.428282 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:58.451591 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:58.488854 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:58.492791 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:58.547825 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:58.549470 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:58.552533 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:58.568208 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:58.569931 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:58.606971 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:58.607075 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:12:58.641501 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:58.675081 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:58.723206 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:58.728294 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:58.743485 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:58.747905 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:58.769596 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:58.772121 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:58.804080 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:58.812536 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:12:58.836144 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:12:58.836272 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:12:58.858129 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:58.859592 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:58.889206 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:12:59.044636 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:12:59.109575 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:59.122082 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:59.130702 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:59.133647 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:59.215457 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:59.235787 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:59.241662 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:59.459277 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:12:59.465775 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:12:59.492436 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:12:59.495518 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:59.549934 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:59.556793 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:59.567925 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:12:59.575482 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:12:59.645223 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:12:59.649848 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:59.721107 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:59.730643 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:59.779422 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:59.807343 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:12:59.814469 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:59.818661 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:59.829913 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:59.854857 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:59.871024 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:12:59.874850 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:59.918913 27374 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:59.918920 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:59.918924 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:59.945066 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:59.950963 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:12:59.984336 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:12:59.994362 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:00.208395 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:00.216908 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:00.219638 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:00.226980 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:00.258834 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:13:00.263731 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:00.279784 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:00.288453 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:00.421226 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:13:00.457079 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:13:00.493247 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:00.515504 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:00.567103 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:00.572183 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:00.580022 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:00.613907 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:00.697672 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:00.698958 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:13:00.721135 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:00.804123 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:13:00.839701 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:00.842139 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:13:00.850121 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:00.851519 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:13:00.871058 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:00.880576 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:00.896936 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:00.927101 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:00.931910 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:00.933033 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:01.068137 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:01.087126 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:01.088019 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:01.089221 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:01.137753 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:01.140532 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:01.170975 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:01.204560 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:13:01.218556 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 20:13:01.318078 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:01.321961 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:01.322801 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:01.333298 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:01.338078 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:13:01.356341 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:01.359068 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:01.388465 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:01.431558 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:13:01.432629 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:13:01.435317 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:13:01.447724 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:13:01.460629 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:01.534092 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:13:01.576571 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:01.597681 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:01.631006 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:01.648437 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:01.694906 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:01.699342 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:01.728608 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:01.895838 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:13:01.946617 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:01.954897 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:01.989696 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:01.998531 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:02.038937 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:02.134812 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:02.135535 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:02.145946 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:02.178567 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:02.179204 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:02.205503 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:02.214529 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:02.224426 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:02.240718 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:02.255867 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:02.256798 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:02.270702 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:02.295624 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:02.312223 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:02.321163 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:02.363763 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:02.370798 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:02.378365 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 44556us
I20240501 20:13:02.420630 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:02.430928 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:02.520567 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:02.586493 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:02.602455 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:02.627926 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:02.629153 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:02.652848 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:02.663668 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:02.673051 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:02.682322 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:02.700327 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:02.978789 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:02.983374 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:02.987445 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:03.047046 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:03.098469 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:03.099366 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:03.106877 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:03.106930 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:03.138723 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:13:03.146824 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:03.163476 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:13:03.227789 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:13:03.266600 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:13:03.283839 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:03.311324 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:03.351629 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:03.353574 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:03.356159 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:03.362190 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:03.392194 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:03.429694 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:03.458972 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:03.461676 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:03.482935 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:03.509511 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:03.531733 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:03.540340 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:03.559669 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:03.566817 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:03.573676 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:03.603245 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:03.641738 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:03.645376 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:03.679630 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:03.703084 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:03.710176 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:03.736515 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:03.757985 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:03.954561 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:03.957125 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:13:03.970602 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:03.985845 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:03.987833 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:13:04.001153 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:04.033139 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:04.035655 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:04.058164 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:04.162034 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:04.198664 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:04.210167 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:04.235314 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:04.250679 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:04.286146 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:04.291128 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:04.296614 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:04.319976 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:04.458194 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:04.516371 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:04.570340 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:04.577894 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:04.587064 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:04.616863 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:04.688213 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:04.698333 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:04.699717 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:04.738538 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:04.857736 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:04.886612 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:04.887991 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:04.921406 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:04.931161 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:04.945688 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:04.967891 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:05.003706 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:05.050807 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:05.114089 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:05.125903 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:05.154601 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:05.188328 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:05.191902 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:05.208355 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:05.211622 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:05.286346 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:13:05.352970 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:05.415529 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:05.434413 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:05.450990 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:05.462833 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:05.523272 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:05.526100 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:05.541388 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:05.542706 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:05.568388 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:05.631208 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:05.674707 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:05.703372 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:05.728323 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:05.741999 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:05.760747 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:05.769755 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:05.770677 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:05.778017 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:05.782270 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:05.782878 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:05.800712 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:13:05.835938 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:05.846916 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:05.855659 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:05.878026 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:05.880956 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:05.897537 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:05.897922 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.075965 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.082468 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:06.099500 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:06.110405 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:06.119051 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.153754 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:06.157464 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:06.167862 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:13:06.283008 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:13:06.294274 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:06.303398 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:13:06.481529 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:13:06.482652 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:13:06.504787 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 20:13:06.545313 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:13:06.547134 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:13:06.703020 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:06.706562 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:06.721381 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.737007 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:06.738869 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.758119 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:06.772873 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.782932 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:06.802600 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:06.830377 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:06.837196 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:06.853806 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:06.858331 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:06.904323 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.912604 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:06.920657 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.969449 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:06.974277 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:06.983029 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:07.004797 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:07.020607 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:07.027283 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:07.056797 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:07.082770 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:07.094451 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:07.104235 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:07.120419 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:07.122849 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:07.130445 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:07.131260 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:07.139137 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:07.228219 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:07.297298 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:13:07.462055 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:13:07.499776 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:07.519106 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:07.560163 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:07.588274 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:07.640961 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:07.649920 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:07.657912 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:07.713017 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:07.736357 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:07.747452 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:07.776675 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:07.795893 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:07.825186 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:07.834069 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:07.853669 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:07.870457 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:07.882839 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:07.918651 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:07.938004 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:07.968047 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 51889us
I20240501 20:13:07.975950 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:07.994056 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:08.007273 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:08.017989 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:08.027590 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:08.043258 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:08.063442 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:08.139271 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:08.215633 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:08.227075 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:08.241348 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:08.244020 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:08.247882 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:08.252367 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:08.264566 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:08.272639 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:13:08.333755 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:08.367046 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:08.638779 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:08.642077 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:08.677450 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:08.692656 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:08.696242 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:08.709503 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:08.728732 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:08.733088 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:08.779903 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:08.794155 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:08.819702 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:08.848557 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:08.868294 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:08.876250 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:08.886096 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:08.922472 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:08.942476 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:08.957998 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:08.991573 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:09.064395 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:09.118119 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:13:09.127308 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:09.127578 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:09.128679 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:09.169909 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:13:09.209654 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:13:09.222858 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:13:09.339036 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:09.358753 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:09.391893 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:09.427606 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:09.456429 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:09.513864 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:09.557893 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:09.560436 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:09.575038 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:09.694365 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:09.699990 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:09.713150 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:09.716909 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:09.721403 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:09.731705 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:09.747102 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:09.771724 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:09.776822 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:09.782742 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:09.784734 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:09.798477 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:09.811404 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:09.839272 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:09.844455 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:09.871451 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:09.890689 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:10.082794 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:10.147320 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:10.172693 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:10.217084 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:10.221140 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:10.227057 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:10.234163 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:10.240447 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:10.316393 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:10.354900 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:10.466190 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:10.502535 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:10.534488 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:10.542574 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:10.598506 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:10.605274 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:10.616061 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:10.634292 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:10.643504 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:13:10.645265 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:13:10.652112 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:10.675251 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:10.676095 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:10.697677 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:10.715610 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:10.731541 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:10.766646 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:10.790932 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:10.798890 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:10.836818 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:10.965809 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:10.972371 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:11.025607 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:11.049785 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:11.071593 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:13:11.076272 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:11.083856 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:11.093822 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:11.227090 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:11.364535 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:11.380554 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:11.399458 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:11.401340 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:11.415635 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 20:13:11.450600 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:11.506599 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:11.531625 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:11.532702 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:13:11.627709 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:13:11.656463 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:13:11.669157 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:13:11.670490 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:13:11.700059 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:13:11.701028 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:13:11.731338 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:11.759222 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:11.775709 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:11.783757 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:11.822237 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:11.847126 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:11.852712 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:11.871582 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 51244.5us
I20240501 20:13:11.886826 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:11.931304 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:11.931710 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:11.971253 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:11.975626 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:12.007478 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:12.019145 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:12.045859 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:12.057873 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:12.153227 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:12.182978 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:12.207715 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:12.220978 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:12.223719 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:12.233140 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:12.263554 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 43892.3us
I20240501 20:13:12.269263 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:12.289868 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:12.303854 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:12.318842 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:13:12.422423 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:12.478394 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:12.490128 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:12.513424 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:12.545550 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:12.546025 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:12.555661 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:12.579488 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:12.760918 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:12.771507 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:12.784034 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:12.832743 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:12.838464 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:12.857060 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:12.858947 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:12.873708 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:12.888541 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:13:12.978724 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:13.127360 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:13.181509 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:13.208467 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:13.226895 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:13.228304 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:13.236452 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:13.257540 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:13.286067 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:13.304348 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:13:13.332301 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:13.359922 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:13.376122 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:13.402333 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:13.421506 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:13.426714 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:13.429754 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:13.440869 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:13.453177 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:13.460568 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:13.609272 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:13.638573 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:13.681504 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:13.688640 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:13.724772 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:13.743034 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:13.778640 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:13.846887 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:13.850536 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:13.857950 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:13.879599 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:13.906930 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:13.916466 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:13.918893 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:13.932507 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:13.969612 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:14.125197 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:13:14.136085 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:13:14.169430 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:13:14.210419 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:14.216614 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:14.233247 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:14.244478 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:14.250154 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:13:14.250830 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:14.286461 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:14.286475 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:14.303195 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:14.342327 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 61846us
I20240501 20:13:14.345286 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:14.402519 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:14.403533 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:14.443552 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:14.456504 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:14.485265 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:14.527776 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:14.551990 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:14.626168 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:14.699317 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:13:14.790632 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:14.807617 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:14.813534 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:14.825021 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:14.825438 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:14.836809 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:14.860924 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:13:14.867452 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:15.112257 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:15.113612 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:15.138267 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:15.151201 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:13:15.168817 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:15.178866 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:15.194888 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:15.195248 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:15.222306 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:15.372884 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:13:15.459517 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:15.472707 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:15.500083 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:15.504763 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:15.533506 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:15.553254 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:15.571203 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:15.587514 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:13:15.608611 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:15.649119 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:15.664227 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:13:15.668848 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:15.670670 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:15.678333 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:13:15.679740 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:15.690128 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:15.716717 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:15.727799 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:15.764222 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:15.768589 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:15.774721 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:15.813982 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:15.820036 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:15.833952 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:15.836843 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:15.837433 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:13:15.859263 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:13:16.077416 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:16.199852 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:16.243912 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:16.260639 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:16.285869 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:16.302887 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:16.357776 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:16.362003 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:16.461383 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:13:16.553967 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:13:16.555289 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:13:16.589252 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:13:16.599845 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:13:16.654855 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:16.666474 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:16.667510 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:16.714104 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:16.714277 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:16.719727 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:16.749053 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240501 20:13:16.768559 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:16.772012 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:16.798712 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:13:16.833017 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:16.847479 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:13:16.891085 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:13:16.896418 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:16.899983 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:16.921308 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:13:16.924207 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:16.965906 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:13:17.014654 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:17.029493 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:17.047370 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:17.098021 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:17.247948 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:17.264387 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:17.311182 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:17.317349 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:17.357957 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:17.359486 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:17.397074 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:17.449648 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:13:17.526559 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:17.572793 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:17.576613 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:17.621304 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:17.627020 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:17.701916 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:17.733620 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:17.734123 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:17.799741 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:13:17.854583 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:13:17.945259 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:17.963352 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:17.975942 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:17.986610 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:18.018983 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:18.041599 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:13:18.067528 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:13:18.081445 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:13:18.107297 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:18.124920 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:18.128712 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:18.154212 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:18.219770 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:18.230891 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:18.242522 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:18.258533 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:18.286484 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:18.303733 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:18.322486 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:18.347272 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:13:18.348434 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:18.366094 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:18.400389 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:18.430868 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:18.447144 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:18.458009 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:18.535881 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:13:18.587335 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:18.609351 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 58296.4us
I20240501 20:13:18.785684 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:18.791401 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:18.791646 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:18.812551 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:18.832262 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:18.854354 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:18.880199 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:18.916743 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:18.942889 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:13:18.957377 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:18.962728 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:18.979725 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:19.000203 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:19.028702 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:19.070412 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:19.075780 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:13:19.149340 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:19.170320 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:19.188899 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:13:19.206560 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:13:19.211561 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:19.254855 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:19.283206 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:19.317160 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:19.354343 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:19.380116 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:19.401964 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:19.490680 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:13:19.515933 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:13:19.546897 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:19.585413 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:19.608448 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:19.715626 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:19.721518 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:19.732096 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:19.749233 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:19.771654 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:19.985469 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:20.083721 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:20.116915 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:20.118458 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:20.122577 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:20.153519 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:20.173825 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:20.176085 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:20.184126 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:20.216758 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:20.221407 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:20.223191 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:20.228304 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:20.229048 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:20.251552 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:20.261070 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:20.278120 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:20.313720 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:20.314378 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:20.331120 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:20.337153 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:20.349908 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:20.357635 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:20.417029 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:20.423532 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:20.478509 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:13:20.478698 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:13:20.562265 27460 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:46843, user_credentials={real_user=slave}} blocked reactor thread for 83584us
W20240501 20:13:20.580729 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:20.706341 26983 maintenance_manager.cc:419] P 423345d7db4f4b2eb388943cdadfc07a: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033382
W20240501 20:13:20.718228 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:20.757174 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:20.767211 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:20.785455 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:20.798481 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:20.807760 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:20.821789 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:20.824855 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:20.861081 26982 maintenance_manager.cc:643] P 423345d7db4f4b2eb388943cdadfc07a: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.154s user 0.034s sys 0.015s Metrics: {"bytes_written":20438,"cfile_init":1,"dirs.queue_time_us":4313,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":857,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1254,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":38333,"threads_started":2,"wal-append.queue_time_us":24722}
I20240501 20:13:20.862895 26983 maintenance_manager.cc:419] P 423345d7db4f4b2eb388943cdadfc07a: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 6767 bytes on disk
I20240501 20:13:20.878721 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:20.889320 26982 maintenance_manager.cc:643] P 423345d7db4f4b2eb388943cdadfc07a: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4}
I20240501 20:13:20.905752 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:21.032881 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:21.044430 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:21.051842 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:21.103104 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:21.119128 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:21.172004 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:21.177609 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:21.193527 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:21.396231 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:21.418440 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:21.428275 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:21.466671 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:21.481554 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:21.489084 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:21.489531 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:21.492949 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:21.505736 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:21.519590 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:21.538966 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:21.570236 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:21.586978 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:21.599886 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 20:13:21.601126 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:13:21.615144 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:21.621315 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:13:21.623181 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:21.641743 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:21.676353 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:21.740453 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:21.766330 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:13:21.795245 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:13:21.989708 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:21.991395 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:21.998838 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:22.011147 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:22.022866 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:22.026337 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:22.051779 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:22.073256 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:22.121340 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:13:22.151738 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:22.155123 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:13:22.217998 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:22.220579 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:22.257908 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:22.308104 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:22.308564 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:22.312721 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:22.322006 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:22.462973 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:22.471648 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:22.494820 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:22.515287 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:13:22.533531 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:22.581004 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:22.589514 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:22.590428 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:22.626211 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:22.626328 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:22.766685 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:22.802896 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:13:22.860498 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:22.868592 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:22.870150 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:22.896996 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:13:22.899015 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:22.914386 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:22.952908 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:22.956995 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:22.985298 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:23.046659 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:23.149785 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:23.154966 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:23.291143 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:23.311777 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:23.316848 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:23.333072 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:23.346921 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:23.381255 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:23.401477 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:23.410907 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:23.439240 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:23.445204 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:23.445904 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:23.471149 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:23.475304 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:23.519990 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:23.655786 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:23.657362 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:23.687144 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:23.707985 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:23.708693 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:23.772567 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:23.779132 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:23.790362 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:23.919893 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:23.931352 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:23.934810 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:23.935209 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:23.966588 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:24.054232 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:24.054595 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:13:24.057911 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:24.067040 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:24.109859 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:13:24.264614 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:24.360459 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:24.370476 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:24.383656 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:24.399621 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:13:24.406759 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:24.434791 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:24.468020 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:24.537362 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:24.567243 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:13:24.578778 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:24.604281 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:24.641685 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:24.653862 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:24.671916 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:24.697408 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:24.736063 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:24.736701 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:24.752540 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:24.770968 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:24.771056 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:24.804730 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:24.837016 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:24.872992 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:24.889436 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:24.889530 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:24.901999 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:24.961241 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:13:25.033342 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:25.092872 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:25.115329 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:25.134857 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:25.138548 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:25.138870 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:25.158438 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:25.194666 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:25.231738 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:13:25.243829 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:25.249595 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:25.421531 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:25.453269 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:25.466864 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:25.516922 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:25.562150 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:25.569844 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:25.575495 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:25.579687 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:25.616791 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:25.715250 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:25.754033 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:25.767494 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:25.780457 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:25.787042 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:25.800772 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:25.810348 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:25.810914 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:25.817618 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:25.835759 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:25.852264 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:25.856516 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:25.880015 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:25.896924 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:25.922504 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:25.936021 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:25.951326 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:26.261786 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:26.276793 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:26.303853 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:26.327622 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:26.339577 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:26.378402 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:26.380931 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:26.396939 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:26.457042 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:13:26.552345 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:26.611994 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:26.638442 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:13:26.665211 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:26.696166 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:26.700642 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:26.720769 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:26.767428 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:26.777796 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:26.846285 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:26.860277 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:13:26.891664 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:26.908545 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:13:26.914266 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:26.955123 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 20:13:26.985224 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:26.988795 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:26.994107 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:13:27.005206 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:27.007375 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:27.008885 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:27.011112 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 58765.6us
W20240501 20:13:27.015542 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:27.034830 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:27.132145 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:27.147306 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:27.154296 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:27.159118 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:27.175271 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:27.198091 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:27.216292 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:27.243307 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:27.282644 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:13:27.313663 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:27.432273 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:27.459232 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:27.476488 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:27.498668 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:27.509805 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:27.519925 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:27.530387 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:27.534230 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:27.573642 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:27.628340 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:27.694485 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:27.848156 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:27.876639 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:27.880551 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:27.899886 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:27.904548 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:27.931890 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:27.940732 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:27.946712 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:27.974647 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:27.976363 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:27.987457 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:28.027922 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:28.068164 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:28.077152 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:28.097514 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:28.143617 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 77481.5us
W20240501 20:13:28.147627 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 83154.6us
I20240501 20:13:28.153704 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:28.166903 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:28.198021 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:28.284505 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:28.434224 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:28.440588 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:28.460007 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:28.481114 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:28.492630 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:28.504487 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:28.520773 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:28.530059 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:28.657018 27200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:43861, user_credentials={real_user=slave}} blocked reactor thread for 59635.9us
W20240501 20:13:28.687577 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:28.701172 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(26cb2329d176441ca9127187db2d157a): perf score=0.033337
I20240501 20:13:28.759495 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:28.781504 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(50bf9058852a4ea49733326057d5ff41): perf score=0.033354
I20240501 20:13:28.794955 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:28.820807 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:28.821053 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:28.860170 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(f02a619bc1584d53ac902c8d8baff976): perf score=0.033375
I20240501 20:13:28.869582 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(50bf9058852a4ea49733326057d5ff41): perf score=0.033379
I20240501 20:13:28.879879 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:28.884490 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:28.906702 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:28.908457 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:28.909314 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:28.925382 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:28.957019 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:29.000496 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:29.051219 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:29.084832 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:29.088120 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(f02a619bc1584d53ac902c8d8baff976) complete. Timing: real 0.227s user 0.057s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":1895,"dirs.queue_time_us":11710,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":6002,"drs_written":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":24300,"threads_started":3,"wal-append.queue_time_us":10611}
I20240501 20:13:29.089757 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(26cb2329d176441ca9127187db2d157a): perf score=0.033437
I20240501 20:13:29.117591 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:29.158113 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:29.217430 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:29.241529 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(26cb2329d176441ca9127187db2d157a) complete. Timing: real 0.146s user 0.052s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2783,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1365,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5880,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":38224,"threads_started":1,"wal-append.queue_time_us":37909}
I20240501 20:13:29.243335 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(f02a619bc1584d53ac902c8d8baff976): 4834 bytes on disk
I20240501 20:13:29.248858 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(f02a619bc1584d53ac902c8d8baff976) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240501 20:13:29.249876 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(26cb2329d176441ca9127187db2d157a): 4581 bytes on disk
I20240501 20:13:29.250433 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(26cb2329d176441ca9127187db2d157a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240501 20:13:29.253073 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:13:29.277213 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:13:29.324131 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:13:29.367592 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(26cb2329d176441ca9127187db2d157a) complete. Timing: real 0.666s user 0.073s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":23472,"dirs.queue_time_us":19501,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":4180,"drs_written":1,"lbm_read_time_us":195,"lbm_reads_lt_1ms":4,"lbm_write_time_us":48903,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":90526,"threads_started":3,"wal-append.queue_time_us":5971}
I20240501 20:13:29.369756 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(26cb2329d176441ca9127187db2d157a): 4581 bytes on disk
I20240501 20:13:29.377539 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(26cb2329d176441ca9127187db2d157a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 20:13:29.382705 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(50bf9058852a4ea49733326057d5ff41) complete. Timing: real 0.512s user 0.086s sys 0.041s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":23406,"dirs.queue_time_us":15960,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":816,"drs_written":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1025,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":52530,"threads_started":3,"wal-append.queue_time_us":9593}
I20240501 20:13:29.384330 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(50bf9058852a4ea49733326057d5ff41): 4922 bytes on disk
I20240501 20:13:29.385133 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(50bf9058852a4ea49733326057d5ff41) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240501 20:13:29.452772 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:13:29.499779 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:29.557986 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:29.595113 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:29.621829 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:29.627522 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:29.628545 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:29.628954 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:29.669389 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:29.720428 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:29.723596 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(50bf9058852a4ea49733326057d5ff41) complete. Timing: real 0.928s user 0.066s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":34303,"dirs.queue_time_us":7458,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1485,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1010,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":92059,"threads_started":3,"wal-append.queue_time_us":13012}
I20240501 20:13:29.726598 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(50bf9058852a4ea49733326057d5ff41): 4922 bytes on disk
I20240501 20:13:29.740450 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(50bf9058852a4ea49733326057d5ff41) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240501 20:13:29.786383 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:29.810883 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:29.822252 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:29.829901 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:29.844852 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:29.888799 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:29.907567 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:29.909354 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:30.003275 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:30.015966 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:13:30.090188 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:30.136266 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:30.168668 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:30.169620 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:30.179878 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:30.195425 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:30.203229 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:30.210191 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:30.251274 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:30.477897 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:13:30.489104 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:13:30.511507 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:30.519655 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:30.589740 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:30.624624 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:30.634436 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:30.658581 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:30.660708 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:30.688208 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:30.721971 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:30.748310 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(553d443dfb344b2690c260d207901ff1): perf score=0.033348
I20240501 20:13:30.756496 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(af693dc19f844b12973b000093021619): perf score=0.033349
W20240501 20:13:30.837113 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:30.897763 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(553d443dfb344b2690c260d207901ff1) complete. Timing: real 0.149s user 0.049s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":24459,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":8712,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":990,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":259968,"thread_start_us":50905,"threads_started":2,"wal-append.queue_time_us":25936}
I20240501 20:13:30.906152 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(553d443dfb344b2690c260d207901ff1): 4834 bytes on disk
I20240501 20:13:30.907292 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(553d443dfb344b2690c260d207901ff1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
W20240501 20:13:30.965615 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:31.024241 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:31.068029 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:31.075770 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:31.088321 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:31.090200 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:31.104930 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:31.113189 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:31.134713 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:31.157812 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:31.191450 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:31.198397 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:31.200012 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:31.234688 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:31.354036 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:31.440955 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:31.566335 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(af693dc19f844b12973b000093021619) complete. Timing: real 0.810s user 0.071s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36013,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":2979,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1069,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":47012,"threads_started":2,"wal-append.queue_time_us":10723}
I20240501 20:13:31.572494 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(af693dc19f844b12973b000093021619): 4921 bytes on disk
W20240501 20:13:31.573452 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:31.573491 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(af693dc19f844b12973b000093021619) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 20:13:31.574438 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(7bb13c1dce8640b28063daa0132d2896): perf score=0.033572
W20240501 20:13:31.586366 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:13:31.689072 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:31.848680 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(7bb13c1dce8640b28063daa0132d2896) complete. Timing: real 0.273s user 0.041s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13798,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":10826,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27603,"threads_started":2,"wal-append.queue_time_us":14541}
I20240501 20:13:31.849869 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(7bb13c1dce8640b28063daa0132d2896): 4581 bytes on disk
W20240501 20:13:31.855930 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 20:13:31.857039 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(7bb13c1dce8640b28063daa0132d2896) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240501 20:13:31.857831 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:13:31.889479 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:13:31.897007 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(309e90bbd9f140a8beca9b3da3f75db4): perf score=0.033365
I20240501 20:13:31.898308 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(78fcaa70e916442ea9c67cc762c8db6d): perf score=0.033365
W20240501 20:13:31.898805 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:31.912901 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(309e90bbd9f140a8beca9b3da3f75db4): perf score=0.033367
I20240501 20:13:31.993177 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:32.007869 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:32.013437 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:32.015532 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:32.022850 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:32.026537 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:13:32.029729 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:32.089754 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:13:32.091117 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:32.099120 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:32.155831 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:32.160225 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:32.162130 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:32.219134 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:32.222194 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:32.230544 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:32.236040 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:32.292991 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:32.349078 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:13:32.419773 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(78fcaa70e916442ea9c67cc762c8db6d) complete. Timing: real 0.521s user 0.058s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12969,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":20232,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":900,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":52238,"threads_started":2,"wal-append.queue_time_us":8803}
I20240501 20:13:32.407716 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(309e90bbd9f140a8beca9b3da3f75db4) complete. Timing: real 0.495s user 0.043s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":26382,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":19755,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":32351,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":99615,"threads_started":2,"wal-append.queue_time_us":21422}
I20240501 20:13:32.425824 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(74c414c2eed045bda9d85fe5c2fea654): perf score=0.033509
I20240501 20:13:32.426757 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(78fcaa70e916442ea9c67cc762c8db6d): perf score=0.033507
I20240501 20:13:32.503263 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(309e90bbd9f140a8beca9b3da3f75db4) complete. Timing: real 0.606s user 0.044s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":24443,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":6778,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":63782,"threads_started":2,"wal-append.queue_time_us":34486}
W20240501 20:13:32.511770 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:13:32.511930 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:32.512817 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(309e90bbd9f140a8beca9b3da3f75db4): 4581 bytes on disk
I20240501 20:13:32.514400 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(309e90bbd9f140a8beca9b3da3f75db4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
W20240501 20:13:32.527168 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:32.536223 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(78fcaa70e916442ea9c67cc762c8db6d) complete. Timing: real 0.109s user 0.071s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1161,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":895,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":25001,"threads_started":1,"wal-append.queue_time_us":25086}
I20240501 20:13:32.538405 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(78fcaa70e916442ea9c67cc762c8db6d): 4834 bytes on disk
I20240501 20:13:32.539052 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(78fcaa70e916442ea9c67cc762c8db6d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 20:13:32.540369 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(309e90bbd9f140a8beca9b3da3f75db4): 4581 bytes on disk
I20240501 20:13:32.540971 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(309e90bbd9f140a8beca9b3da3f75db4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 20:13:32.621920 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(74c414c2eed045bda9d85fe5c2fea654) complete. Timing: real 0.193s user 0.039s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1734,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":908,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11890,"threads_started":1,"wal-append.queue_time_us":11957}
I20240501 20:13:32.625052 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(74c414c2eed045bda9d85fe5c2fea654): 4921 bytes on disk
I20240501 20:13:32.689481 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(74c414c2eed045bda9d85fe5c2fea654) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240501 20:13:32.692126 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(78fcaa70e916442ea9c67cc762c8db6d): 4834 bytes on disk
W20240501 20:13:32.700804 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:32.723937 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(78fcaa70e916442ea9c67cc762c8db6d) complete. Timing: real 0.030s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240501 20:13:32.748608 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:13:32.769392 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:32.805475 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:32.809376 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:32.857442 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:32.876848 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:32.885262 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:32.897658 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:32.917868 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:32.973909 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:32.977849 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:33.044251 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(b5427fbc8a244a69974fd8d0f5f3d08d): perf score=0.033342
I20240501 20:13:33.055585 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:33.076939 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:13:33.077014 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:33.096792 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:33.104568 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:33.152350 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:33.182540 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:33.186898 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:33.224881 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:33.227389 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(17e69dbd245e4b8db397aaffc9145c9f): perf score=0.033394
W20240501 20:13:33.232362 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:33.268210 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(17e69dbd245e4b8db397aaffc9145c9f): perf score=0.033393
I20240501 20:13:33.289222 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:33.307144 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:33.332379 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:33.418664 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:33.419335 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:33.448289 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:33.467710 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(b5427fbc8a244a69974fd8d0f5f3d08d) complete. Timing: real 0.423s user 0.063s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":145167,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":3567,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4367,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148321,"threads_started":2,"wal-append.queue_time_us":3342}
I20240501 20:13:33.470983 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(b5427fbc8a244a69974fd8d0f5f3d08d): 4921 bytes on disk
I20240501 20:13:33.471715 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(b5427fbc8a244a69974fd8d0f5f3d08d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
W20240501 20:13:33.479173 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 87916.3us
I20240501 20:13:33.492780 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(17e69dbd245e4b8db397aaffc9145c9f) complete. Timing: real 0.265s user 0.042s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":36730,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":8012,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":925,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":39478,"threads_started":2,"wal-append.queue_time_us":2450}
I20240501 20:13:33.500706 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(b5427fbc8a244a69974fd8d0f5f3d08d): perf score=0.033457
I20240501 20:13:33.504478 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:33.584293 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(17e69dbd245e4b8db397aaffc9145c9f) complete. Timing: real 0.316s user 0.101s sys 0.066s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2496,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":43195,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":956,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":133305,"threads_started":2,"wal-append.queue_time_us":129432}
I20240501 20:13:33.585711 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(17e69dbd245e4b8db397aaffc9145c9f): 4834 bytes on disk
I20240501 20:13:33.586279 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(17e69dbd245e4b8db397aaffc9145c9f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 20:13:33.592516 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(b5427fbc8a244a69974fd8d0f5f3d08d) complete. Timing: real 0.088s user 0.036s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1158,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":984,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10249,"threads_started":1,"wal-append.queue_time_us":10135}
I20240501 20:13:33.593870 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(b5427fbc8a244a69974fd8d0f5f3d08d): 4921 bytes on disk
I20240501 20:13:33.599289 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(b5427fbc8a244a69974fd8d0f5f3d08d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240501 20:13:33.602591 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(17e69dbd245e4b8db397aaffc9145c9f): 4834 bytes on disk
I20240501 20:13:33.605881 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(17e69dbd245e4b8db397aaffc9145c9f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":180,"lbm_reads_lt_1ms":4}
W20240501 20:13:33.693600 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:33.733387 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:33.759574 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:33.770756 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:33.778672 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:33.801281 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:33.809731 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:33.811136 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:33.852298 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:33.954775 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:33.957927 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:33.982024 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:34.004570 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:34.046212 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:34.058072 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:34.060838 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:34.061287 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:34.088860 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(b5a20d47052f404986315ddb05284ad9): perf score=0.033336
I20240501 20:13:34.120687 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(c0e88937e1ec49bfbaffe535e445e40a): perf score=0.033348
I20240501 20:13:34.227241 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(3f8902b7b0b649edb10f362713ad36fe): perf score=0.033381
I20240501 20:13:34.255358 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(c0e88937e1ec49bfbaffe535e445e40a) complete. Timing: real 0.134s user 0.062s sys 0.040s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31641,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":77059,"threads_started":2,"wal-append.queue_time_us":43045}
I20240501 20:13:34.256635 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(c0e88937e1ec49bfbaffe535e445e40a): 4581 bytes on disk
I20240501 20:13:34.257257 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(c0e88937e1ec49bfbaffe535e445e40a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 20:13:34.284235 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(b5a20d47052f404986315ddb05284ad9) complete. Timing: real 0.195s user 0.064s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8653,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2965,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":755,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":7424,"thread_start_us":13308,"threads_started":2,"wal-append.queue_time_us":4618}
I20240501 20:13:34.286846 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(b5a20d47052f404986315ddb05284ad9): 4834 bytes on disk
I20240501 20:13:34.287540 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(b5a20d47052f404986315ddb05284ad9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240501 20:13:34.305078 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:34.333750 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:34.367130 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(3f8902b7b0b649edb10f362713ad36fe) complete. Timing: real 0.140s user 0.054s sys 0.049s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":46206,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":2890,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":56570,"threads_started":2,"wal-append.queue_time_us":10589}
I20240501 20:13:34.369486 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(3f8902b7b0b649edb10f362713ad36fe): 4921 bytes on disk
I20240501 20:13:34.371806 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(3f8902b7b0b649edb10f362713ad36fe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240501 20:13:34.372145 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:13:34.395538 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:34.421049 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:34.421746 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:34.436141 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:34.436841 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:34.446864 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:34.452188 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:34.453886 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:34.483386 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:13:34.519923 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:34.614909 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:34.618454 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:34.638859 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:34.660645 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:34.660728 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:34.709478 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:34.719494 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:34.738440 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:34.827772 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:34.981071 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:34.995149 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:35.010916 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:35.023008 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:35.043833 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:35.059142 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:35.075073 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:35.103477 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:35.112480 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:35.115747 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:35.156504 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:35.160804 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:35.182307 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:35.210322 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:35.232087 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:35.329907 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:35.351143 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:35.365648 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:35.392510 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:35.438644 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:35.454443 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:35.470294 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:35.541205 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:35.570924 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:35.587385 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:35.594868 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:35.618410 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:13:35.643555 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:13:35.687940 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:35.802953 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:35.819332 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:35.836484 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:35.865931 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:35.886334 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:35.914721 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:35.921483 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:35.948660 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:36.019054 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(6c836d9afa0f439b978bc2fe5b5e499d): perf score=0.033338
I20240501 20:13:36.046497 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(8bf74c24f28546bb90e10002270e3814): perf score=0.033352
W20240501 20:13:36.082075 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:36.133397 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(6c836d9afa0f439b978bc2fe5b5e499d) complete. Timing: real 0.114s user 0.034s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3088,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":2377,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6244,"threads_started":2,"wal-append.queue_time_us":2928}
I20240501 20:13:36.135602 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(8bf74c24f28546bb90e10002270e3814): perf score=0.033377
I20240501 20:13:36.136922 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(6c836d9afa0f439b978bc2fe5b5e499d): 4581 bytes on disk
I20240501 20:13:36.138206 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(6c836d9afa0f439b978bc2fe5b5e499d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 20:13:36.235615 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:36.248121 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:36.274374 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(8bf74c24f28546bb90e10002270e3814) complete. Timing: real 0.223s user 0.033s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":28450,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":2445,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":92633,"threads_started":2,"wal-append.queue_time_us":21279}
I20240501 20:13:36.279364 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(c619a2235c9248eeb6b2e1d78583d1ba): perf score=0.033400
I20240501 20:13:36.305397 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(8bf74c24f28546bb90e10002270e3814) complete. Timing: real 0.170s user 0.040s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":340,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1635,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":778,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":27518,"threads_started":2,"wal-append.queue_time_us":15712}
I20240501 20:13:36.308032 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(8bf74c24f28546bb90e10002270e3814): 4834 bytes on disk
I20240501 20:13:36.308868 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(8bf74c24f28546bb90e10002270e3814) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 20:13:36.313351 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:36.340943 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:36.364018 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:36.394443 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:36.450517 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:36.465276 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:36.481458 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:36.487044 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:36.491530 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:36.505617 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:36.520347 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:36.564695 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:36.569278 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:36.590680 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:36.667176 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:13:36.718107 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:36.718410 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:36.725850 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:36.745379 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:36.747664 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(c619a2235c9248eeb6b2e1d78583d1ba) complete. Timing: real 0.468s user 0.058s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":21115,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":9877,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9740,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":22049,"threads_started":2,"wal-append.queue_time_us":1121}
I20240501 20:13:36.749296 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(c619a2235c9248eeb6b2e1d78583d1ba): 4921 bytes on disk
I20240501 20:13:36.751837 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(c619a2235c9248eeb6b2e1d78583d1ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240501 20:13:36.759806 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(8bf74c24f28546bb90e10002270e3814): 4834 bytes on disk
I20240501 20:13:36.761485 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(8bf74c24f28546bb90e10002270e3814) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":154,"lbm_reads_lt_1ms":4}
I20240501 20:13:36.769733 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:36.782142 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:36.808697 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:36.825070 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:36.843436 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:13:36.902700 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:13:36.925762 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:36.987838 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:37.019454 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(d2160f17e62749dbb41405a895516b06): perf score=0.033338
I20240501 20:13:37.067044 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(d2160f17e62749dbb41405a895516b06): perf score=0.033360
I20240501 20:13:37.089649 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:37.087538 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(d2160f17e62749dbb41405a895516b06) complete. Timing: real 0.068s user 0.040s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1097,"drs_written":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2231,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11383,"threads_started":1,"wal-append.queue_time_us":11487}
I20240501 20:13:37.092715 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(d2160f17e62749dbb41405a895516b06): 4581 bytes on disk
I20240501 20:13:37.103107 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(d2160f17e62749dbb41405a895516b06) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 20:13:37.117326 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:37.118438 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:13:37.122370 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(9721fc9afa4741b78823e293514e6265): perf score=0.033369
I20240501 20:13:37.142618 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(9721fc9afa4741b78823e293514e6265): perf score=0.033375
I20240501 20:13:37.155090 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:37.161073 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:37.201465 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:37.207605 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:37.240128 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:37.246184 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:13:37.272670 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:37.286520 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:37.311875 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:37.344516 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:37.344280 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(d2160f17e62749dbb41405a895516b06) complete. Timing: real 0.277s user 0.071s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":70930,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":3772,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":780,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":88283,"threads_started":2,"wal-append.queue_time_us":5233}
I20240501 20:13:37.351213 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(d2160f17e62749dbb41405a895516b06): 4581 bytes on disk
I20240501 20:13:37.351891 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(d2160f17e62749dbb41405a895516b06) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240501 20:13:37.353232 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(e5b9f1a087114c82b010bd10e4f676c1): perf score=0.033433
W20240501 20:13:37.361636 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:37.376111 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:37.404038 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:37.436744 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:37.438441 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(9721fc9afa4741b78823e293514e6265) complete. Timing: real 0.313s user 0.058s sys 0.038s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39050,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":2409,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":977,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":244608,"thread_start_us":90895,"threads_started":2,"wal-append.queue_time_us":52020}
I20240501 20:13:37.439882 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(e5b9f1a087114c82b010bd10e4f676c1): perf score=0.033452
W20240501 20:13:37.463795 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:13:37.497207 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:37.501271 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(e5b9f1a087114c82b010bd10e4f676c1) complete. Timing: real 0.148s user 0.035s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8229,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1222,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1158,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8719,"threads_started":1,"wal-append.queue_time_us":8614}
I20240501 20:13:37.505388 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(e5b9f1a087114c82b010bd10e4f676c1): 4834 bytes on disk
I20240501 20:13:37.506597 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(e5b9f1a087114c82b010bd10e4f676c1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 20:13:37.519197 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:37.532660 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(e5b9f1a087114c82b010bd10e4f676c1) complete. Timing: real 0.093s user 0.046s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1152,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2327,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6398,"threads_started":1,"wal-append.queue_time_us":6298}
I20240501 20:13:37.534085 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(9721fc9afa4741b78823e293514e6265): 4921 bytes on disk
I20240501 20:13:37.536242 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(9721fc9afa4741b78823e293514e6265) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 20:13:37.537369 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(e5b9f1a087114c82b010bd10e4f676c1): 4834 bytes on disk
I20240501 20:13:37.537904 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(e5b9f1a087114c82b010bd10e4f676c1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240501 20:13:37.540695 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(9721fc9afa4741b78823e293514e6265) complete. Timing: real 0.398s user 0.047s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":32554,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":4818,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1008,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":122754,"threads_started":2,"wal-append.queue_time_us":19159}
I20240501 20:13:37.541946 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(9721fc9afa4741b78823e293514e6265): 4921 bytes on disk
I20240501 20:13:37.544617 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(9721fc9afa4741b78823e293514e6265) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240501 20:13:37.614959 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:13:37.776103 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:37.779273 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:37.792896 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:37.797688 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:37.804800 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:37.827471 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:37.860086 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:37.860556 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:37.886298 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:37.918505 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:37.948770 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:37.968345 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:38.008764 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:38.028389 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:38.080293 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:38.097573 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:38.105201 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:38.128974 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:38.141099 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:13:38.213224 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:13:38.310858 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:38.312947 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:38.335891 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:38.337540 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:38.349174 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:38.385095 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:38.419829 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:38.427446 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:38.444278 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:38.626816 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:38.677388 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:38.681691 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:38.685315 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:38.704087 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:38.707391 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:38.710642 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:38.757571 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:13:38.786921 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 55170.9us
I20240501 20:13:38.895988 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:38.900296 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:38.912127 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:38.947338 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:38.968367 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:38.971316 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:38.982108 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:38.999598 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:39.017740 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:39.078619 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:13:39.117339 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(944d354124664bc0870f251d8a0831c5): perf score=0.033350
W20240501 20:13:39.264005 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:39.265980 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(4dc5794a67374cc388da2edc47cbb113): perf score=0.033390
I20240501 20:13:39.283653 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(944d354124664bc0870f251d8a0831c5) complete. Timing: real 0.166s user 0.055s sys 0.042s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14838,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1595,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1142,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":7424,"thread_start_us":22083,"threads_started":2,"wal-append.queue_time_us":7251}
I20240501 20:13:39.285029 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(944d354124664bc0870f251d8a0831c5): 4834 bytes on disk
I20240501 20:13:39.286705 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(944d354124664bc0870f251d8a0831c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 20:13:39.289768 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:39.305205 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(944d354124664bc0870f251d8a0831c5): perf score=0.033396
I20240501 20:13:39.322449 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:39.346700 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:39.352221 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:39.420873 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:39.429217 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:39.457513 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(944d354124664bc0870f251d8a0831c5) complete. Timing: real 0.152s user 0.053s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2058,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":4872,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1116,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":5823,"threads_started":2,"wal-append.queue_time_us":3432}
I20240501 20:13:39.470046 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(944d354124664bc0870f251d8a0831c5): 4834 bytes on disk
I20240501 20:13:39.473027 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(944d354124664bc0870f251d8a0831c5) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":173,"lbm_reads_lt_1ms":4}
I20240501 20:13:39.476630 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:39.492543 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:39.498158 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:39.535308 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(4dc5794a67374cc388da2edc47cbb113) complete. Timing: real 0.269s user 0.044s sys 0.042s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":96442,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":11236,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":875,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":118122,"threads_started":2,"wal-append.queue_time_us":13890}
I20240501 20:13:39.537807 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(e7f03e02d13f44d188e5eb2774771a0d): perf score=0.033464
W20240501 20:13:39.618417 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:13:39.676236 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:39.690012 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:39.704838 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:39.717576 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:39.718516 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:39.720909 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(e7f03e02d13f44d188e5eb2774771a0d) complete. Timing: real 0.183s user 0.036s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1214,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":33453,"threads_started":1,"wal-append.queue_time_us":33349}
I20240501 20:13:39.724479 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(4dc5794a67374cc388da2edc47cbb113): 4921 bytes on disk
I20240501 20:13:39.727279 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(4dc5794a67374cc388da2edc47cbb113) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240501 20:13:39.728375 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(e7f03e02d13f44d188e5eb2774771a0d): 4581 bytes on disk
I20240501 20:13:39.728922 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(e7f03e02d13f44d188e5eb2774771a0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 20:13:39.742748 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:39.756310 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:39.793251 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:39.799366 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:39.869658 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:39.879029 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:39.882342 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:39.894584 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:39.894702 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:39.934015 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:39.952672 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:39.961027 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:13:39.987859 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:13:40.014562 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:40.227805 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(00e3777032eb4d9c80f47855e5f9f87a): perf score=0.033353
I20240501 20:13:40.234884 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(5272c34458af43bebb5ba5baefac7137): perf score=0.033355
W20240501 20:13:40.266258 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:40.292177 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(ebcdab26b4ed45f596282d4805d1bb1f): perf score=0.033371
I20240501 20:13:40.299960 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(ebcdab26b4ed45f596282d4805d1bb1f): perf score=0.033380
I20240501 20:13:40.302999 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:40.345031 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:40.369443 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:40.400358 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:40.428716 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:40.439365 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:40.439554 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:40.440812 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:40.463303 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(ebcdab26b4ed45f596282d4805d1bb1f) complete. Timing: real 0.171s user 0.059s sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35849,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":4842,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":952,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":96696,"threads_started":2,"wal-append.queue_time_us":61019}
I20240501 20:13:40.466203 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(ebcdab26b4ed45f596282d4805d1bb1f): 4581 bytes on disk
I20240501 20:13:40.466956 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(ebcdab26b4ed45f596282d4805d1bb1f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240501 20:13:40.475850 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:40.521169 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:40.529007 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:40.526398 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(00e3777032eb4d9c80f47855e5f9f87a) complete. Timing: real 0.298s user 0.080s sys 0.069s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":106459,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1433,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":214264,"threads_started":2,"wal-append.queue_time_us":96684}
I20240501 20:13:40.538004 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(00e3777032eb4d9c80f47855e5f9f87a): 4834 bytes on disk
I20240501 20:13:40.538616 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(00e3777032eb4d9c80f47855e5f9f87a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 20:13:40.553455 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:40.570849 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:40.603008 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:40.620867 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:40.622156 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:40.633664 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:40.725077 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(ebcdab26b4ed45f596282d4805d1bb1f) complete. Timing: real 0.425s user 0.061s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":28186,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":69991,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11082,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":37521,"threads_started":2,"wal-append.queue_time_us":3539}
I20240501 20:13:40.727012 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(5272c34458af43bebb5ba5baefac7137): perf score=0.033493
I20240501 20:13:40.795073 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(5272c34458af43bebb5ba5baefac7137) complete. Timing: real 0.560s user 0.046s sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":16288,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":2860,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":56034,"threads_started":2,"wal-append.queue_time_us":29295}
I20240501 20:13:40.796332 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(5272c34458af43bebb5ba5baefac7137): 4922 bytes on disk
I20240501 20:13:40.805711 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(5272c34458af43bebb5ba5baefac7137) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240501 20:13:40.828176 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:13:40.830519 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:40.856026 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:40.864126 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(5272c34458af43bebb5ba5baefac7137) complete. Timing: real 0.134s user 0.054s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":255,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1253,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":800,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":35679,"threads_started":1,"wal-append.queue_time_us":35793}
I20240501 20:13:40.865448 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(5272c34458af43bebb5ba5baefac7137): 4922 bytes on disk
I20240501 20:13:40.875293 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(5272c34458af43bebb5ba5baefac7137) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":167,"lbm_reads_lt_1ms":4}
I20240501 20:13:40.876575 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(ebcdab26b4ed45f596282d4805d1bb1f): 4581 bytes on disk
I20240501 20:13:40.877116 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(ebcdab26b4ed45f596282d4805d1bb1f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 20:13:40.892908 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:40.911087 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:40.937625 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:40.951380 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:40.962738 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:40.964004 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:41.051688 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:13:41.064599 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:41.244766 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:41.276494 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:41.285807 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:41.316069 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:41.331066 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:41.349188 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:41.384354 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:41.413450 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:41.474882 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling FlushMRSOp(57b4bd50bc8f4d50957bd0280224c420): perf score=0.033358
I20240501 20:13:41.484094 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:41.488821 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:41.524500 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:41.544294 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling FlushMRSOp(0c836bb145a54160966c475e1490da38): perf score=0.033377
I20240501 20:13:41.563123 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling FlushMRSOp(c2432c8e8e8847fa977055be919e79d6): perf score=0.033385
I20240501 20:13:41.576500 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:41.577164 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:13:41.581714 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:41.631050 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(c2432c8e8e8847fa977055be919e79d6): perf score=0.033398
I20240501 20:13:41.644861 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:41.707517 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:41.708405 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:41.712401 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: FlushMRSOp(57b4bd50bc8f4d50957bd0280224c420) complete. Timing: real 0.237s user 0.043s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35551,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":6024,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":956,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60760,"threads_started":2,"wal-append.queue_time_us":17738}
I20240501 20:13:41.713961 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(57b4bd50bc8f4d50957bd0280224c420): 4581 bytes on disk
I20240501 20:13:41.715122 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(57b4bd50bc8f4d50957bd0280224c420) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
W20240501 20:13:41.841284 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:41.845546 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: FlushMRSOp(0c836bb145a54160966c475e1490da38) complete. Timing: real 0.301s user 0.065s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7661,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":7875,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14536,"threads_started":2,"wal-append.queue_time_us":6627}
I20240501 20:13:41.849855 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(0c836bb145a54160966c475e1490da38): 4921 bytes on disk
I20240501 20:13:41.850452 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(0c836bb145a54160966c475e1490da38) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240501 20:13:41.856686 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: FlushMRSOp(c2432c8e8e8847fa977055be919e79d6) complete. Timing: real 0.293s user 0.063s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":20546,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":10755,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":856,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":60824,"threads_started":2,"wal-append.queue_time_us":25242}
I20240501 20:13:41.858579 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(c2432c8e8e8847fa977055be919e79d6): 4834 bytes on disk
I20240501 20:13:41.862435 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(c2432c8e8e8847fa977055be919e79d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240501 20:13:41.919824 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(c2432c8e8e8847fa977055be919e79d6) complete. Timing: real 0.289s user 0.055s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":27038,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":10120,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":69578,"threads_started":2,"wal-append.queue_time_us":23927}
I20240501 20:13:41.921362 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(c2432c8e8e8847fa977055be919e79d6): 4834 bytes on disk
W20240501 20:13:41.923172 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:41.926347 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(c2432c8e8e8847fa977055be919e79d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240501 20:13:41.927521 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling FlushMRSOp(57b4bd50bc8f4d50957bd0280224c420): perf score=0.033476
I20240501 20:13:42.005437 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: FlushMRSOp(57b4bd50bc8f4d50957bd0280224c420) complete. Timing: real 0.078s user 0.042s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1163,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":945,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1972,"threads_started":1,"wal-append.queue_time_us":2063}
I20240501 20:13:42.007503 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(57b4bd50bc8f4d50957bd0280224c420): 4581 bytes on disk
I20240501 20:13:42.008391 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(57b4bd50bc8f4d50957bd0280224c420) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240501 20:13:42.040128 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:42.040864 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:42.058852 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:42.084954 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:42.089123 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:42.102548 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:42.109089 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:42.119277 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:42.123152 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:42.151578 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:13:42.187793 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:42.198770 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:42.220125 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:42.237185 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:42.249164 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:42.287389 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:42.326989 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:42.354637 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 20:13:42.379168 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:42.379599 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:13:42.389293 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:42.382563 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:13:42.396219 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:13:42.555019 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:42.567265 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:13:42.584818 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:42.587723 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:42.597873 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:42.604022 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:42.613662 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:42.614346 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:42.618155 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:42.676662 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:42.857311 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:42.909478 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:42.926244 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:42.926811 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:42.933254 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:42.976330 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:43.002101 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:43.067248 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:43.069578 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:43.086298 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:43.105679 27202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 63277.4us
I20240501 20:13:43.199811 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:43.256261 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:43.276278 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:43.322525 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:43.368561 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:43.372298 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:43.388968 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:43.399055 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:43.440919 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:43.473682 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:43.484689 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:43.517346 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:43.563958 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:43.564323 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:43.573844 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:43.583113 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:43.588075 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:43.606570 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:43.798365 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:43.813439 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:43.835309 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:43.853156 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:43.889950 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:43.902202 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:43.924700 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:43.951808 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:44.074817 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:44.092190 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:44.097888 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:44.119272 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:13:44.166030 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:44.195384 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:44.201344 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:44.201540 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:44.219592 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:44.288198 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:13:44.471235 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:44.481781 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:44.499197 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:44.515532 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:44.515625 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:44.533063 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:44.560674 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:44.562981 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:44.674160 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:13:44.735980 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:13:44.759073 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:13:44.778704 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:13:44.800251 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:44.837308 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:44.855937 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:44.857004 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:13:44.871142 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:44.892521 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:44.893893 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:44.925527 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:44.931381 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:45.032091 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:13:45.035473 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:13:45.059919 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:45.070935 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:45.086879 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:45.088766 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:45.108727 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:45.126186 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:45.129616 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:45.247401 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:13:45.322669 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:45.344445 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:45.344673 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:45.346060 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:45.372104 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:45.410660 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:45.412127 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:45.457558 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:45.533097 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:13:45.612797 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:13:45.684537 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:45.693199 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:45.699988 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:45.743289 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:13:45.750157 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:45.775198 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:45.786741 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:45.799487 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:45.850368 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 71044.8us
I20240501 20:13:46.014272 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:46.016671 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:46.035972 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:46.073686 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:46.090842 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:46.091662 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:46.104573 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:46.132658 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:46.142930 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:46.156666 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:46.162053 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:13:46.168805 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:46.191951 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:46.202304 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:46.222955 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:46.276705 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:46.412336 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:46.511193 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:46.513219 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:46.520582 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:46.523847 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:46.525417 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:46.544636 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:46.589985 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:46.605379 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 70931.3us
W20240501 20:13:46.652814 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:13:46.667019 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:46.732941 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:46.740370 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:46.766240 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:46.767282 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:46.797314 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:46.798089 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:46.811188 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:46.834669 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:13:47.010202 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:47.082685 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:47.200743 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:47.219312 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:47.241582 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:47.261113 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 20:13:47.273900 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:47.293606 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:47.327087 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:13:47.331281 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:47.374943 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:47.391088 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:13:47.446803 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:13:47.449635 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:47.461019 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:47.474256 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:47.506264 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:47.506507 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:47.513773 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:47.515434 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:47.538143 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:47.549988 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:47.563830 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:47.570528 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:47.585345 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:13:47.593919 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:47.594836 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:47.612398 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:47.630167 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:47.702389 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:47.716359 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:47.719157 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:47.724665 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:47.752362 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:47.764348 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:47.788180 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:13:47.802142 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:47.806492 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:47.822247 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:47.835373 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:47.845646 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:47.851066 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:47.921819 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:13:47.975064 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:48.132678 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:48.141377 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:48.152827 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:48.164038 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:48.178295 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:48.189548 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:48.190248 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:48.193189 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:48.254426 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:13:48.382045 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:48.596165 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:48.614198 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:48.631330 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:48.648969 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:48.692186 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:48.695987 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:48.729147 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:48.751364 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:13:48.758931 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:48.767554 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:48.789871 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:48.807477 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:48.823606 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:48.838250 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:48.839056 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:48.862354 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:48.895413 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:48.924925 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:48.942576 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:49.007606 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:49.048533 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:49.130805 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:49.169698 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:49.172168 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:49.177554 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:13:49.219929 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:13:49.268114 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:13:49.362190 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:49.366014 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:49.366446 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:49.412537 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:49.440951 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:49.469308 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:49.547601 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:49.576661 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:49.615922 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:49.616331 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:49.617966 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:49.633174 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:49.691901 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:49.696043 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:49.713593 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:13:49.725996 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:49.772452 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:13:49.781258 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:13:49.796859 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:13:49.813814 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:13:49.975543 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:13:50.027220 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:50.050001 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:50.073854 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:50.114691 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:50.120039 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:13:50.134159 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:50.169781 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:13:50.178994 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:50.203531 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:50.251799 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:50.316144 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:13:50.421994 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:50.453334 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:50.469669 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:50.509521 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:50.528959 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:50.531935 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:50.532835 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:50.561291 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:50.576189 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:50.594282 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:50.606935 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:50.616294 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:50.624868 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:50.642792 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:50.659278 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:13:50.728075 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:13:50.771423 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:50.873172 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:50.873593 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:50.875892 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:50.909792 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:50.966742 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:50.967264 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:50.967429 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:51.110190 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:51.134475 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:51.138888 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:51.153563 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:13:51.156100 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:51.172920 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:51.199842 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:51.231053 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:51.270251 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:51.462515 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:51.520781 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:51.578511 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(4dc5794a67374cc388da2edc47cbb113): perf score=0.033366
I20240501 20:13:51.610373 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:51.631811 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:51.655927 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:13:51.657531 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:51.660082 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:51.680593 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:51.696254 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:51.696866 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:13:51.711427 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:51.711587 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:51.712774 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:51.749179 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:51.772137 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:51.785460 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:51.810317 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:51.839027 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:51.858899 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:51.859262 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:51.879392 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:51.934863 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:51.956544 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:51.986671 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:52.011194 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 139792us
I20240501 20:13:52.057303 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:52.082537 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:13:52.199846 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:52.236655 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(4dc5794a67374cc388da2edc47cbb113) complete. Timing: real 0.657s user 0.074s sys 0.034s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":53766,"dirs.queue_time_us":56812,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":2576,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12709,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":140406,"threads_started":3,"wal-append.queue_time_us":20511}
I20240501 20:13:52.243818 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(4dc5794a67374cc388da2edc47cbb113): 4921 bytes on disk
I20240501 20:13:52.245404 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(4dc5794a67374cc388da2edc47cbb113) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 20:13:52.246521 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(78fcaa70e916442ea9c67cc762c8db6d): perf score=0.033496
W20240501 20:13:52.280916 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:13:52.282210 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:13:52.297904 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:52.331624 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(78fcaa70e916442ea9c67cc762c8db6d) complete. Timing: real 0.084s user 0.056s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1294,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4794,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7117,"threads_started":1,"wal-append.queue_time_us":7222}
I20240501 20:13:52.333328 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(78fcaa70e916442ea9c67cc762c8db6d): 4834 bytes on disk
I20240501 20:13:52.334172 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(78fcaa70e916442ea9c67cc762c8db6d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240501 20:13:52.347189 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:52.370287 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:13:52.537679 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:13:52.546670 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:52.548488 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 20:13:52.549048 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:52.558898 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:52.596262 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(b5427fbc8a244a69974fd8d0f5f3d08d): perf score=0.033368
W20240501 20:13:52.607818 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:13:52.609493 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:13:52.610527 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:52.611457 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:52.638852 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:52.696478 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:52.716756 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:52.731314 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(309e90bbd9f140a8beca9b3da3f75db4): perf score=0.033380
I20240501 20:13:52.754144 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:52.773229 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:52.825896 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:52.851730 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:52.851788 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:52.856969 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:13:52.857916 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:52.899633 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:52.907546 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:52.925091 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:53.061095 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:53.071439 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:53.087035 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:53.104182 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:53.105517 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:53.107546 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:53.117110 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:53.157028 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:53.169333 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:53.257710 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(b5427fbc8a244a69974fd8d0f5f3d08d) complete. Timing: real 0.661s user 0.055s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":34012,"dirs.queue_time_us":16862,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":6586,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1720,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":76394,"threads_started":3,"wal-append.queue_time_us":18134}
I20240501 20:13:53.272295 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(b5427fbc8a244a69974fd8d0f5f3d08d): 4921 bytes on disk
I20240501 20:13:53.275547 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(b5427fbc8a244a69974fd8d0f5f3d08d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240501 20:13:53.278086 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(b5a20d47052f404986315ddb05284ad9): perf score=0.033481
W20240501 20:13:53.353423 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:53.471805 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(b5a20d47052f404986315ddb05284ad9) complete. Timing: real 0.181s user 0.039s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1299,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10806,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8442,"threads_started":1,"wal-append.queue_time_us":8386}
I20240501 20:13:53.475739 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(b5a20d47052f404986315ddb05284ad9): 4834 bytes on disk
I20240501 20:13:53.476452 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(b5a20d47052f404986315ddb05284ad9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 20:13:53.478058 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(6c836d9afa0f439b978bc2fe5b5e499d): perf score=0.033462
I20240501 20:13:53.505528 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(309e90bbd9f140a8beca9b3da3f75db4) complete. Timing: real 0.771s user 0.061s sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":29722,"dirs.queue_time_us":9064,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":3433,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1009,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":87822,"threads_started":3,"wal-append.queue_time_us":49987}
I20240501 20:13:53.522572 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(8bf74c24f28546bb90e10002270e3814): perf score=0.033484
I20240501 20:13:53.572731 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:53.591203 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(b5a20d47052f404986315ddb05284ad9): perf score=0.033402
I20240501 20:13:53.593297 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(6c836d9afa0f439b978bc2fe5b5e499d) complete. Timing: real 0.115s user 0.036s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1889,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1309,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":11520,"thread_start_us":10483,"threads_started":1,"wal-append.queue_time_us":10526}
I20240501 20:13:53.599833 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(6c836d9afa0f439b978bc2fe5b5e499d): 4581 bytes on disk
I20240501 20:13:53.600462 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(6c836d9afa0f439b978bc2fe5b5e499d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 20:13:53.630415 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:53.684827 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:53.684926 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:53.695541 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:53.742317 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:53.753697 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:53.808403 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:53.863922 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:53.886467 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(8bf74c24f28546bb90e10002270e3814) complete. Timing: real 0.363s user 0.045s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11238,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":6049,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4866,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16105,"threads_started":2,"wal-append.queue_time_us":4603}
I20240501 20:13:53.888674 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(8bf74c24f28546bb90e10002270e3814): 4834 bytes on disk
I20240501 20:13:53.889873 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(8bf74c24f28546bb90e10002270e3814) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 20:13:53.899601 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(309e90bbd9f140a8beca9b3da3f75db4): 4581 bytes on disk
I20240501 20:13:53.900297 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(309e90bbd9f140a8beca9b3da3f75db4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 20:13:53.901638 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(b5427fbc8a244a69974fd8d0f5f3d08d): perf score=0.033581
W20240501 20:13:53.989467 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:54.042271 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(b5a20d47052f404986315ddb05284ad9) complete. Timing: real 0.448s user 0.077s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":446174,"compiler_manager_pool.run_cpu_time_us":67,"compiler_manager_pool.run_wall_time_us":67,"dirs.queue_time_us":9772,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":5200,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":33749,"threads_started":2,"wal-append.queue_time_us":23690}
I20240501 20:13:54.044314 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(57b4bd50bc8f4d50957bd0280224c420): perf score=0.033472
W20240501 20:13:54.053871 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:54.083596 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(b5427fbc8a244a69974fd8d0f5f3d08d) complete. Timing: real 0.180s user 0.066s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1081,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1106,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":21200,"threads_started":1,"wal-append.queue_time_us":21309}
I20240501 20:13:54.085157 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(b5427fbc8a244a69974fd8d0f5f3d08d): 4921 bytes on disk
I20240501 20:13:54.085865 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(b5427fbc8a244a69974fd8d0f5f3d08d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 20:13:54.110772 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(f02a619bc1584d53ac902c8d8baff976): perf score=0.033365
W20240501 20:13:54.197360 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:13:54.308728 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(57b4bd50bc8f4d50957bd0280224c420) complete. Timing: real 0.264s user 0.041s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1219,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1047,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":14281,"threads_started":1,"wal-append.queue_time_us":14165}
I20240501 20:13:54.311825 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(f034a679870b44caa238f04a8b49d302): perf score=0.033535
I20240501 20:13:54.372989 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(f02a619bc1584d53ac902c8d8baff976) complete. Timing: real 0.261s user 0.074s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37250,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":4590,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2612,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":10112,"thread_start_us":45955,"threads_started":2,"wal-append.queue_time_us":8678}
I20240501 20:13:54.377584 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(f02a619bc1584d53ac902c8d8baff976): 4834 bytes on disk
I20240501 20:13:54.378697 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(f02a619bc1584d53ac902c8d8baff976) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240501 20:13:54.399451 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 76144.3us
W20240501 20:13:54.403662 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:54.411840 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:54.490454 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:54.491453 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:54.540238 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:54.553256 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:54.553256 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:54.561214 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:54.587365 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:54.588652 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(c0e88937e1ec49bfbaffe535e445e40a): perf score=0.033344
I20240501 20:13:54.609766 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:54.631441 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(74c414c2eed045bda9d85fe5c2fea654): perf score=0.033342
I20240501 20:13:54.674221 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:54.684769 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:54.693634 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:54.750397 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(c0e88937e1ec49bfbaffe535e445e40a) complete. Timing: real 0.162s user 0.058s sys 0.083s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45346,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":3109,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1180,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":99726,"threads_started":2,"wal-append.queue_time_us":54336}
I20240501 20:13:54.750936 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:54.751832 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(c0e88937e1ec49bfbaffe535e445e40a): 4581 bytes on disk
I20240501 20:13:54.752569 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(c0e88937e1ec49bfbaffe535e445e40a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240501 20:13:54.763231 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:54.785535 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:54.809463 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:54.825628 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:13:54.843712 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:13:54.867668 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:54.906973 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(f034a679870b44caa238f04a8b49d302) complete. Timing: real 0.595s user 0.043s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":50212,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":5861,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":915,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25472,"threads_started":2,"wal-append.queue_time_us":309}
I20240501 20:13:54.909344 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(b5a20d47052f404986315ddb05284ad9): 4834 bytes on disk
I20240501 20:13:54.910161 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(b5a20d47052f404986315ddb05284ad9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
I20240501 20:13:54.911850 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(57b4bd50bc8f4d50957bd0280224c420): 4581 bytes on disk
I20240501 20:13:54.913756 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(57b4bd50bc8f4d50957bd0280224c420) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 20:13:54.915585 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(f034a679870b44caa238f04a8b49d302): 4581 bytes on disk
I20240501 20:13:54.918401 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(f034a679870b44caa238f04a8b49d302) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240501 20:13:54.919869 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:13:54.922329 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:13:54.923270 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:54.945714 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:54.958279 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:54.959271 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:54.990442 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:54.993508 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:54.994395 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:55.004410 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(9721fc9afa4741b78823e293514e6265): perf score=0.033342
I20240501 20:13:55.043444 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:55.088599 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(74c414c2eed045bda9d85fe5c2fea654) complete. Timing: real 0.457s user 0.084s sys 0.038s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":67464,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":3106,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1365,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":187590,"threads_started":2,"wal-append.queue_time_us":80297}
I20240501 20:13:55.110294 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(74c414c2eed045bda9d85fe5c2fea654): 4921 bytes on disk
I20240501 20:13:55.111004 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(74c414c2eed045bda9d85fe5c2fea654) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240501 20:13:55.163723 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 46916.6us
W20240501 20:13:55.183565 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:13:55.190019 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:55.274868 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:55.283178 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:55.311386 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:55.314378 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:55.336900 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(9721fc9afa4741b78823e293514e6265) complete. Timing: real 0.332s user 0.062s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37955,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1424,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":839,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":53821,"threads_started":2,"wal-append.queue_time_us":9533}
I20240501 20:13:55.351234 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:55.356796 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(9721fc9afa4741b78823e293514e6265): 4921 bytes on disk
I20240501 20:13:55.357714 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(9721fc9afa4741b78823e293514e6265) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240501 20:13:55.359344 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:55.363205 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(3f8902b7b0b649edb10f362713ad36fe): perf score=0.033351
I20240501 20:13:55.363338 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(af693dc19f844b12973b000093021619): perf score=0.033370
I20240501 20:13:55.402197 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:55.411497 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:55.463959 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(af693dc19f844b12973b000093021619) complete. Timing: real 0.100s user 0.045s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1286,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1409,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":44241,"threads_started":1,"wal-append.queue_time_us":44334}
I20240501 20:13:55.465345 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(af693dc19f844b12973b000093021619): 4921 bytes on disk
I20240501 20:13:55.466033 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(af693dc19f844b12973b000093021619) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240501 20:13:55.492997 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:55.509959 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(3f8902b7b0b649edb10f362713ad36fe) complete. Timing: real 0.146s user 0.044s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1111,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":847,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":18048,"threads_started":1,"wal-append.queue_time_us":18151}
I20240501 20:13:55.513355 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(3f8902b7b0b649edb10f362713ad36fe): 4921 bytes on disk
I20240501 20:13:55.513984 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(3f8902b7b0b649edb10f362713ad36fe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240501 20:13:55.596378 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:55.598132 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:55.603021 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:55.674549 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(c619a2235c9248eeb6b2e1d78583d1ba): perf score=0.033382
I20240501 20:13:55.710465 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:55.754987 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:55.755971 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:55.768334 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(af693dc19f844b12973b000093021619): perf score=0.033347
I20240501 20:13:55.775615 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:55.776791 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:55.796432 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:56.000581 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(c619a2235c9248eeb6b2e1d78583d1ba) complete. Timing: real 0.326s user 0.088s sys 0.031s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":27735,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":8902,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1066,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":68594,"threads_started":2,"wal-append.queue_time_us":41940}
I20240501 20:13:56.014856 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(af693dc19f844b12973b000093021619) complete. Timing: real 0.246s user 0.071s sys 0.038s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1166,"drs_written":1,"lbm_read_time_us":582,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":43446,"threads_started":1,"wal-append.queue_time_us":43574}
I20240501 20:13:56.017124 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(af693dc19f844b12973b000093021619): 4921 bytes on disk
I20240501 20:13:56.018468 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(af693dc19f844b12973b000093021619) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 20:13:56.029928 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(e7f03e02d13f44d188e5eb2774771a0d): perf score=0.033445
I20240501 20:13:56.125969 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:56.134017 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:56.169063 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:56.220904 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:56.241511 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:56.271121 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:56.283421 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:56.286089 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:56.343144 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:56.346385 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:56.359479 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:56.366127 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:56.370095 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:56.395992 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:56.422333 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:56.431334 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:13:56.454401 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:56.506253 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(e7f03e02d13f44d188e5eb2774771a0d) complete. Timing: real 0.476s user 0.036s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":1785,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":19270,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":47265,"threads_started":1,"wal-append.queue_time_us":47374}
I20240501 20:13:56.508728 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(c619a2235c9248eeb6b2e1d78583d1ba): 4921 bytes on disk
I20240501 20:13:56.509589 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(c619a2235c9248eeb6b2e1d78583d1ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 20:13:56.511129 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(e7f03e02d13f44d188e5eb2774771a0d): 4581 bytes on disk
I20240501 20:13:56.511668 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(e7f03e02d13f44d188e5eb2774771a0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240501 20:13:56.570060 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:56.581131 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:56.609716 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:56.632486 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:56.722468 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(57b4bd50bc8f4d50957bd0280224c420): perf score=0.033348
W20240501 20:13:56.739135 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:56.765620 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(00e3777032eb4d9c80f47855e5f9f87a): perf score=0.033387
W20240501 20:13:56.771845 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:13:56.834072 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:56.840029 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:56.842577 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:56.855450 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:56.951485 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(00e3777032eb4d9c80f47855e5f9f87a) complete. Timing: real 0.186s user 0.057s sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":53257,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":4058,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":771,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":69408,"threads_started":2,"wal-append.queue_time_us":2735}
I20240501 20:13:56.954209 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(00e3777032eb4d9c80f47855e5f9f87a): 4834 bytes on disk
I20240501 20:13:56.955032 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(00e3777032eb4d9c80f47855e5f9f87a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240501 20:13:56.970129 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:56.973800 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:56.979916 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:57.008680 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:57.028510 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:57.034231 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(553d443dfb344b2690c260d207901ff1): perf score=0.033396
I20240501 20:13:57.045205 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:57.054695 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:57.056751 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:57.058426 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(57b4bd50bc8f4d50957bd0280224c420) complete. Timing: real 0.336s user 0.085s sys 0.025s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":93366,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":60173,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":149899,"threads_started":2,"wal-append.queue_time_us":55734}
I20240501 20:13:57.067785 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(57b4bd50bc8f4d50957bd0280224c420): 4581 bytes on disk
I20240501 20:13:57.068586 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(57b4bd50bc8f4d50957bd0280224c420) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240501 20:13:57.128553 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:13:57.134510 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:57.222134 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(553d443dfb344b2690c260d207901ff1) complete. Timing: real 0.188s user 0.040s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":34125,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1685,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":796,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":57660,"threads_started":2,"wal-append.queue_time_us":23722}
I20240501 20:13:57.234455 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(553d443dfb344b2690c260d207901ff1): 4834 bytes on disk
I20240501 20:13:57.235248 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(553d443dfb344b2690c260d207901ff1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4}
I20240501 20:13:57.242872 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:13:57.270610 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:57.324239 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:13:57.368409 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:57.465162 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:57.475653 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:57.480201 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 20:13:57.481099 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:13:57.482091 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:57.494901 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:57.508327 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:57.509544 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:57.530781 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:57.548943 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:57.557410 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:57.567838 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:57.590703 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:57.596680 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:57.616583 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:57.622309 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:57.629818 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:13:57.651087 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:13:57.673789 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:13:57.724848 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:13:57.785646 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:13:57.964167 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:57.997035 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:58.001209 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:58.010063 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:58.052323 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:58.061463 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:58.074584 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:58.105067 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:13:58.106135 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:58.109828 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:58.145635 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:58.157752 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:58.199067 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:58.201172 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:58.215246 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(3f8902b7b0b649edb10f362713ad36fe): perf score=0.033399
W20240501 20:13:58.228533 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:58.243160 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:58.247990 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:58.260512 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:58.405500 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(3f8902b7b0b649edb10f362713ad36fe) complete. Timing: real 0.189s user 0.060s sys 0.033s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":29001,"dirs.run_cpu_time_us":310,"dirs.run_wall_time_us":5093,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2633,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":35390,"threads_started":2,"wal-append.queue_time_us":6392}
I20240501 20:13:58.407430 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(3f8902b7b0b649edb10f362713ad36fe): 4921 bytes on disk
I20240501 20:13:58.408193 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(3f8902b7b0b649edb10f362713ad36fe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 20:13:58.499104 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:58.500140 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:58.502355 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:58.537158 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:58.563786 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:58.566828 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:58.568161 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:58.580639 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(50bf9058852a4ea49733326057d5ff41): perf score=0.033351
I20240501 20:13:58.659984 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(5272c34458af43bebb5ba5baefac7137): perf score=0.033346
I20240501 20:13:58.743923 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(c2432c8e8e8847fa977055be919e79d6): perf score=0.033338
W20240501 20:13:58.849651 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:58.850661 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:58.867086 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:58.876709 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:58.885931 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(c2432c8e8e8847fa977055be919e79d6) complete. Timing: real 0.142s user 0.049s sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6832,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":7918,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1410,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":52791,"threads_started":2,"wal-append.queue_time_us":45827}
I20240501 20:13:58.887907 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(c2432c8e8e8847fa977055be919e79d6): 4834 bytes on disk
I20240501 20:13:58.888514 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:58.889816 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(c2432c8e8e8847fa977055be919e79d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 20:13:58.890376 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:58.913604 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:13:58.915525 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(5272c34458af43bebb5ba5baefac7137) complete. Timing: real 0.255s user 0.028s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1694,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1384,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1006,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15932,"threads_started":1,"wal-append.queue_time_us":16217}
I20240501 20:13:58.930984 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(5272c34458af43bebb5ba5baefac7137): 4922 bytes on disk
I20240501 20:13:58.931579 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(5272c34458af43bebb5ba5baefac7137) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 20:13:58.932762 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(50bf9058852a4ea49733326057d5ff41): perf score=0.033349
I20240501 20:13:58.990386 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:59.002611 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:59.062958 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(50bf9058852a4ea49733326057d5ff41) complete. Timing: real 0.130s user 0.042s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14017,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1303,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":869,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":32512,"thread_start_us":7224,"threads_started":1,"wal-append.queue_time_us":7139}
I20240501 20:13:59.064386 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(50bf9058852a4ea49733326057d5ff41): 4922 bytes on disk
I20240501 20:13:59.064980 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(50bf9058852a4ea49733326057d5ff41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 20:13:59.069033 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(50bf9058852a4ea49733326057d5ff41) complete. Timing: real 0.486s user 0.053s sys 0.039s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39330,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":2190,"drs_written":1,"lbm_read_time_us":1488,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":894,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71019,"threads_started":2,"wal-append.queue_time_us":31388}
I20240501 20:13:59.070215 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(50bf9058852a4ea49733326057d5ff41): 4922 bytes on disk
I20240501 20:13:59.074151 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(50bf9058852a4ea49733326057d5ff41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
W20240501 20:13:59.132819 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:13:59.193388 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:13:59.326227 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(ebcdab26b4ed45f596282d4805d1bb1f): perf score=0.033345
W20240501 20:13:59.332721 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:13:59.341081 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:13:59.362427 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:59.362841 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:59.377056 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:13:59.379961 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:13:59.384577 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:59.422395 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:59.440181 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:59.445065 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:13:59.453974 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:59.466405 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(ebcdab26b4ed45f596282d4805d1bb1f) complete. Timing: real 0.140s user 0.035s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1365,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":969,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2239,"threads_started":1,"wal-append.queue_time_us":2138}
I20240501 20:13:59.483882 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:59.485097 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(ebcdab26b4ed45f596282d4805d1bb1f): 4581 bytes on disk
I20240501 20:13:59.485792 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:59.487417 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(ebcdab26b4ed45f596282d4805d1bb1f) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 20:13:59.515285 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:13:59.534428 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:13:59.551867 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:13:59.556185 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:59.582947 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:13:59.602322 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:13:59.624411 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:13:59.624851 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:13:59.898409 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(d2160f17e62749dbb41405a895516b06): perf score=0.033338
I20240501 20:13:59.920725 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:13:59.951071 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:13:59.958884 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:13:59.975792 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:13:59.998450 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:00.040064 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:00.046509 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:00.060820 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:00.068864 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:14:00.070670 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:00.079922 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(d2160f17e62749dbb41405a895516b06) complete. Timing: real 0.181s user 0.036s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10162,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":2345,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3977,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":8960,"thread_start_us":12682,"threads_started":2,"wal-append.queue_time_us":2531}
I20240501 20:14:00.082058 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(d2160f17e62749dbb41405a895516b06): 4581 bytes on disk
W20240501 20:14:00.088127 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:14:00.089720 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:00.102792 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:14:00.103129 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:00.105495 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(d2160f17e62749dbb41405a895516b06) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 20:14:00.171475 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:00.172435 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:00.188154 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:00.244625 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:00.311558 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:00.344736 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:00.384353 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:00.388888 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:00.396960 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:00.404100 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:14:00.411821 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:14:00.412856 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:00.422955 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:00.435890 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:00.628620 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:00.671809 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:00.695746 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:00.700850 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:00.716794 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:00.761024 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:00.783555 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:00.836944 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:00.853315 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:00.898329 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:00.926409 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:00.961721 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:00.996157 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:01.000259 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:01.017517 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:01.062824 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:14:01.301653 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:01.331197 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:01.333890 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:01.347011 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:01.364012 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:01.365463 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(0c836bb145a54160966c475e1490da38): perf score=0.033390
I20240501 20:14:01.419469 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:01.438727 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:01.475536 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:01.495337 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(0c836bb145a54160966c475e1490da38) complete. Timing: real 0.130s user 0.072s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9413,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":5868,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1188,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":18081,"threads_started":2,"wal-append.queue_time_us":8584}
I20240501 20:14:01.497208 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(0c836bb145a54160966c475e1490da38): 4921 bytes on disk
I20240501 20:14:01.497952 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(0c836bb145a54160966c475e1490da38) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 20:14:01.572573 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:01.602607 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:01.607476 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:14:01.615993 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:01.646405 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:14:01.648078 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:01.661556 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:01.687443 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:01.704643 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:01.725909 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:14:01.736768 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:14:01.762166 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 114810us
W20240501 20:14:01.797576 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:14:01.878592 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:14:01.893527 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:01.916822 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:14:01.924167 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:01.928576 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:01.957162 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:01.996686 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:02.006235 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:02.071190 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:02.081959 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:02.085677 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:02.087749 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:02.133185 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:02.144397 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:02.176992 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:14:02.213527 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:02.237504 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:14:02.239229 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:02.265129 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:02.468351 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:02.480789 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:02.486804 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 20:14:02.488016 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:02.493791 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:02.525108 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:02.559758 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:14:02.576270 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:14:02.580601 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:02.595584 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:14:02.617869 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:02.626996 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:02.779428 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:02.825307 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:02.832841 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:02.837740 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:14:02.852093 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:14:02.871366 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:02.890288 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:02.907078 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:02.913841 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:14:02.947353 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:02.957330 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:03.080374 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:14:03.107188 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:14:03.234289 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:03.247059 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:03.286478 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:03.288595 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:03.317819 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:03.345014 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:03.361763 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:03.384052 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:03.394472 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:03.400550 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:03.405230 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:03.418743 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:03.419253 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:03.445705 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:03.463822 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:03.512418 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:14:03.705461 29271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 81706.6us
I20240501 20:14:03.835608 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:03.868182 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:03.877338 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:03.888296 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:03.900874 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:03.977459 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:14:03.987545 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:04.000138 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:04.010349 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:04.157680 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:14:04.221115 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:04.229744 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:04.247543 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:04.285082 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:04.289297 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:14:04.291064 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:04.331323 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:04.338140 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:04.346783 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:14:04.377543 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:04.386368 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:04.391392 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:04.426640 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:04.450753 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 111283us
W20240501 20:14:04.456528 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 135328us
I20240501 20:14:04.468072 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:04.493894 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:04.501896 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:14:04.515885 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:04.518849 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:04.520156 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:04.634933 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:04.684540 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:14:04.708016 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:04.736449 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:04.745133 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:04.760776 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:14:04.782070 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:14:04.782960 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:04.811800 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:04.857935 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:04.913964 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:05.007486 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:05.012513 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:05.028231 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:05.080204 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:05.086185 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:05.130450 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:05.186760 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:05.208859 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:05.228327 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:05.257944 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:05.260366 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:05.283555 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:05.296275 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:05.299253 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:05.362236 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:05.386255 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:05.420147 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:05.432694 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:05.476286 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:05.541649 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:05.593592 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:05.610704 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:05.627638 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:05.639721 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:05.680526 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:05.706420 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:05.726404 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:05.940706 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:05.946616 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:05.961568 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:05.978214 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:06.043516 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:06.048094 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:06.055040 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:06.067214 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:06.206809 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:14:06.379267 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:06.396086 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:06.398317 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:06.415674 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:06.433099 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:06.445115 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:06.484277 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:06.489800 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:06.542618 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:06.546974 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:14:06.568439 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:06.581733 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:06.635003 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 81027.3us
I20240501 20:14:06.637454 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:06.638080 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:06.715546 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:06.725952 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:14:06.742041 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:06.743418 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:06.801306 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:06.839663 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:06.863756 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:14:06.865145 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:06.942920 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:06.955027 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:06.955430 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:06.959262 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:07.023305 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:14:07.029673 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:07.063524 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:07.135809 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:14:07.213485 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:14:07.268832 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:07.271860 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:14:07.284880 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:14:07.285548 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:07.291684 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:07.307065 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:07.340484 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:07.423851 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:07.438652 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:07.444001 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:07.510841 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:07.515887 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:07.524091 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:07.541581 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:07.562279 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:14:07.592798 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:14:07.593003 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:07.593470 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:07.602901 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:07.676349 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 20:14:07.712148 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:07.715075 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:07.733507 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:07.733783 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:07.736440 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:07.771219 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:07.798547 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:14:07.802294 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:07.951912 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:07.957545 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 20:14:07.981397 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:14:08.072499 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:08.081984 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:14:08.094364 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:08.157949 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:08.166415 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:08.179616 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:08.227298 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:08.230744 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:08.364820 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:08.474238 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:14:08.531296 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:08.570137 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:08.593658 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:08.629786 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:08.648955 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:08.650297 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:08.668996 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:08.671706 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:08.692564 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:08.709631 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:08.719465 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:08.733363 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:08.765192 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:08.772454 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:08.851066 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:14:08.922394 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:08.969578 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:08.985891 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:08.993824 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:09.009023 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:09.030557 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:09.035982 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:09.054039 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:09.061887 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:09.134301 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:14:09.175001 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:14:09.176205 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:14:09.253793 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 48941us
I20240501 20:14:09.275152 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:09.327049 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:09.351126 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:09.354241 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:09.391716 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:09.392576 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:09.417627 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:09.453846 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:09.482373 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:09.507815 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:09.586954 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:14:09.637688 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:14:09.707193 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:14:09.743137 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:09.869699 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:09.875835 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:09.890930 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:09.904592 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:09.919081 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:09.993311 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:09.999785 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:10.010890 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:10.016685 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:10.019383 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:10.048173 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:10.051980 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:10.084940 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:10.093096 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:10.116705 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:10.147727 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:10.174023 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:10.187661 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:10.192214 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:10.246985 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:10.262130 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:14:10.299909 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:10.313199 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:10.319118 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:10.334359 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:10.520884 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:10.570994 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:10.591107 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:10.595443 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:10.615388 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:10.688011 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:10.702683 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:10.744009 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:10.746471 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:10.759310 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:10.775213 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:10.791723 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:10.802919 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:10.867599 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:10.868994 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:10.884079 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:10.885741 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:11.195439 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:11.215672 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:11.230713 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:11.261333 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:11.264858 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:11.295619 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:11.312114 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:11.320236 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:11.340232 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:11.403852 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:11.405371 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:11.407039 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:11.429854 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:11.481063 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:11.516481 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:11.519780 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:11.570466 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:14:11.606734 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:14:11.650580 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:14:11.657090 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:11.660144 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:11.677793 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:11.722018 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:11.756422 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:11.771234 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:11.785403 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:11.793761 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:11.847404 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:14:11.857420 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:14:11.993083 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:14:12.045105 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:12.081140 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:12.085619 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:12.088240 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:12.095412 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 47212.9us
I20240501 20:14:12.129387 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:12.193164 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:12.211748 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:12.222879 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:14:12.223271 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:14:12.227854 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:12.240913 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:12.272025 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:12.278581 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:12.280736 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:12.291636 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:12.309039 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:12.332944 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:12.349059 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:14:12.378175 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:14:12.440094 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:14:12.536859 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:12.579053 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:12.581760 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:12.598208 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:12.614463 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:12.703752 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:12.720172 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:12.727500 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:14:12.772480 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:12.815306 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:12.851197 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:12.876626 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:12.895450 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:12.900506 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:14:12.949649 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:12.973181 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:13.003576 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:13.077927 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:14:13.088951 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:14:13.109145 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:14:13.217741 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:14:13.355577 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:13.373828 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:13.388600 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:13.421877 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:13.435343 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:13.441360 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:13.451467 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:13.476702 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:13.535554 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:13.557318 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:14:13.559764 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 48251us
I20240501 20:14:13.577239 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:13.588456 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:13.625633 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:13.636400 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:13.763137 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:13.776623 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:13.854535 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 79762.8us
W20240501 20:14:13.871883 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:13.872622 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:14:13.920938 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:13.986876 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:14.000200 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:14.004595 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:14.008369 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:14.015007 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:14.042892 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:14.053596 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:14.072016 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:14.114328 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:14.148721 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:14.168988 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:14.176533 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:14.256583 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:14.264010 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:14:14.264181 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:14:14.273417 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:14.277746 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:14.295336 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:14:14.326436 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:14.553987 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:14.564325 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:14.577811 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:14.599400 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:14.628789 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:14.645279 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:14.669970 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:14.685524 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:14.697584 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:14.705780 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:14.720489 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:14.735936 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:14.740427 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:14.744230 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:14:14.750298 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:14.825709 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:14:14.831995 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:14.840051 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:14.852414 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:14.860476 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:14.878710 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:15.024395 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:15.034296 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:15.038923 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:15.042759 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:15.066118 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(7bb13c1dce8640b28063daa0132d2896): perf score=0.033389
I20240501 20:14:15.081702 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:15.095785 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:15.112900 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:15.167629 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:14:15.175288 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:15.184049 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 57137.3us
I20240501 20:14:15.284343 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(7bb13c1dce8640b28063daa0132d2896) complete. Timing: real 0.218s user 0.040s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5511,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":2070,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":962,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11807,"threads_started":2,"wal-append.queue_time_us":6297}
I20240501 20:14:15.291818 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(7bb13c1dce8640b28063daa0132d2896): 4581 bytes on disk
I20240501 20:14:15.292596 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(7bb13c1dce8640b28063daa0132d2896) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 20:14:15.367574 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:15.398108 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:15.398258 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:15.417831 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:15.421685 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:15.429020 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:15.429473 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:14:15.444346 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:15.498770 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:15.702463 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:15.728266 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:15.761309 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:15.775835 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:15.786307 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:15.796860 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:15.814456 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:15.843365 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:15.846330 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:15.848678 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:15.849627 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:15.914724 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:15.932838 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:15.962849 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:15.976145 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:16.241633 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:16.266047 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:16.319167 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:16.327011 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:16.356922 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:16.364409 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:16.377801 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:16.391896 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:16.507236 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:16.584271 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(4dc5794a67374cc388da2edc47cbb113): perf score=0.033366
W20240501 20:14:16.595891 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:14:16.634292 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:16.698881 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:16.722999 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:16.747735 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:14:16.756846 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:16.778410 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:16.783998 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:16.784494 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:16.824920 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:16.833882 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:16.913887 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:16.913743 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(4dc5794a67374cc388da2edc47cbb113) complete. Timing: real 0.329s user 0.065s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60080,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":9189,"drs_written":1,"lbm_read_time_us":6483,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1041,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":63406,"threads_started":2,"wal-append.queue_time_us":3043}
I20240501 20:14:16.915693 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(4dc5794a67374cc388da2edc47cbb113): 4921 bytes on disk
I20240501 20:14:16.921824 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(4dc5794a67374cc388da2edc47cbb113) complete. Timing: real 0.005s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 20:14:16.929260 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:16.930919 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:16.935470 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:16.960696 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:16.968895 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:16.997103 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:17.007766 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 98132.1us
W20240501 20:14:17.029491 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:14:17.030665 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:17.133281 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:14:17.136727 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:14:17.154379 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:17.182947 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:17.201156 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:17.217979 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:17.230273 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:17.231858 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:17.265869 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:17.323379 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:17.374378 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:14:17.397780 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:14:17.446200 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:17.455924 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:17.462688 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:14:17.473655 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:17.489121 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:17.511060 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:17.521236 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:17.539134 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:17.601815 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:14:17.602119 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:17.787346 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:14:17.848019 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:14:17.858817 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:14:17.926985 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:17.933137 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:17.950560 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:17.952854 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:17.973347 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:17.985777 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:18.015900 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:18.023429 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:18.067555 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:18.082670 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:18.089507 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:18.138402 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:18.139972 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 20:14:18.168780 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:18.182818 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(553d443dfb344b2690c260d207901ff1): perf score=0.033375
I20240501 20:14:18.208395 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:18.227264 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:18.354282 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(553d443dfb344b2690c260d207901ff1) complete. Timing: real 0.171s user 0.048s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":65315,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":11376,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1756,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":117567,"threads_started":2,"wal-append.queue_time_us":28946}
I20240501 20:14:18.356061 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(553d443dfb344b2690c260d207901ff1): 4834 bytes on disk
I20240501 20:14:18.356696 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(553d443dfb344b2690c260d207901ff1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240501 20:14:18.386533 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:18.541867 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:14:18.557932 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:18.616888 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:18.659854 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:18.674326 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:18.689136 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:18.702373 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:14:18.704912 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:18.711508 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:18.716830 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:18.757982 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:18.780267 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:18.785651 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:18.789587 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:18.802105 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:18.818821 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:18.837826 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:18.980429 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:14:19.013253 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:14:19.041954 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:19.060743 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:19.061715 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:19.061987 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:19.075346 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:14:19.136374 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:19.138572 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:19.149521 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:19.282406 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:14:19.345172 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:14:19.400147 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:19.406258 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:19.441934 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:19.463150 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:19.485343 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:14:19.517086 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:19.559230 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:19.593329 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:14:19.597549 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:19.599591 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:19.687741 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:14:19.756682 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:14:19.842545 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:14:19.846449 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:19.878705 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:19.900882 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:19.901382 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:19.902079 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:19.933976 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:19.948944 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:19.951812 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:19.957604 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:14:19.976070 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:19.984133 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:20.016212 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:20.025363 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:20.031952 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:20.043534 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:20.072863 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:20.172382 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:14:20.266477 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:14:20.287722 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:20.328948 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:14:20.364522 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:20.371132 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:20.395310 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:20.400295 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:14:20.401438 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:20.412642 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:20.474700 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:20.481463 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:20.830547 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:20.849349 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:20.858548 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:20.872921 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:20.881947 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:20.911397 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:20.926455 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:20.976132 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 144943us
I20240501 20:14:20.983587 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:21.031925 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:21.042568 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:21.092027 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:21.108196 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:21.148553 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:21.161144 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:21.195829 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:21.329506 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:21.418356 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:21.432335 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:21.499544 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:21.500305 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:21.508134 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:21.510991 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:21.514236 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:21.528350 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:21.536705 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:21.576138 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:21.576422 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:21.595280 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:21.598230 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:21.608861 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:14:21.622658 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:21.664006 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:21.691169 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:14:21.698828 27068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 82823.1us
W20240501 20:14:21.713075 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:14:21.738366 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:21.793213 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:14:21.818887 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:14:21.884945 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:14:21.930400 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:21.933274 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:21.945062 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:21.958961 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:21.977357 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:21.998049 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:22.023806 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:22.223114 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:14:22.244439 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:22.276687 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:22.290814 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:22.296358 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:22.314179 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:22.332963 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:22.349104 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:22.352878 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:22.374207 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:22.378692 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:22.383209 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(f02a619bc1584d53ac902c8d8baff976): perf score=0.033365
W20240501 20:14:22.454103 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 90343.7us
W20240501 20:14:22.472689 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:22.570808 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(f02a619bc1584d53ac902c8d8baff976) complete. Timing: real 0.187s user 0.050s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":16677,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1604,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":768,"thread_start_us":37382,"threads_started":2,"wal-append.queue_time_us":20949}
I20240501 20:14:22.576893 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(f02a619bc1584d53ac902c8d8baff976): 4834 bytes on disk
I20240501 20:14:22.577558 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(f02a619bc1584d53ac902c8d8baff976) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240501 20:14:22.588268 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:22.598093 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:22.630260 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:22.666781 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:22.722873 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:22.761639 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:22.761952 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:22.784198 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:22.796195 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:22.809826 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:22.838104 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:22.847970 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:14:22.932829 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:14:22.970160 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:23.010684 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:23.033452 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:23.050931 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:23.051385 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:23.053195 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:23.101789 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:23.229984 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 20:14:23.313997 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:23.352766 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:23.353404 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:23.393211 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:23.403393 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:23.448290 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:23.469316 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:23.495260 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:23.581396 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:23.596765 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:23.600365 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:14:23.623816 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:23.687980 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:23.715317 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:23.750564 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:23.754285 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:23.877652 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:14:23.924381 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:23.959003 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:23.964401 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:23.966984 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:23.981490 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:23.987771 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:23.991376 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:24.022939 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:24.028872 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:24.054503 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:14:24.138239 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:24.143604 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:24.182291 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:24.191954 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:24.208437 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:24.209632 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:24.212399 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:24.227432 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:24.228206 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:24.280910 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:24.328109 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:24.332051 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:14:24.423266 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:24.449944 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:24.451560 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:24.468261 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:24.480197 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:24.481069 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:24.483704 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:24.533071 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:24.578341 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:24.591228 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:24.592340 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:24.625301 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:24.645790 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:24.683259 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:24.690521 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:24.704473 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:24.714576 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:24.762285 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:24.799249 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:14:24.865936 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:25.009032 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:25.038142 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:25.109012 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:25.145448 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:25.182278 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:25.196919 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:25.201167 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:25.202173 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:25.220410 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:25.222656 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:25.241035 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:25.255304 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:25.258975 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:25.284339 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:25.293740 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:25.353231 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:25.369848 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:25.550945 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:25.570032 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:14:25.591619 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:25.595165 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:25.610668 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:25.620113 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:25.622898 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:25.639175 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:25.679011 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:25.801376 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:25.819836 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:25.825070 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:25.833586 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:25.836078 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:25.853514 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:25.855944 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:25.927750 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:26.123102 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:26.124830 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:26.138347 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:26.155426 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:26.203624 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:26.263568 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 102392us
I20240501 20:14:26.307540 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:26.319622 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:26.325736 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:26.336100 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:26.341145 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:26.375854 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:26.413378 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:26.443871 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:26.444007 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:26.472944 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:26.509012 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:14:26.600803 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:14:26.646517 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 66923.3us
W20240501 20:14:26.661118 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:26.675902 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:26.700269 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:26.701267 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:26.708791 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 61186.5us
I20240501 20:14:26.719942 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:26.738864 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:26.794967 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:26.806697 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:26.823333 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:26.831842 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:26.869777 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:26.884063 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:26.954103 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:26.954550 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:14:26.969868 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:26.984531 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:26.999179 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:27.018945 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:27.028298 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:27.047468 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:27.047797 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:27.085361 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:14:27.131595 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:27.311633 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:27.374682 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:27.390116 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:27.410846 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:27.474381 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:27.498651 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:27.500079 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:27.520486 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:27.553563 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:27.579224 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:14:27.579594 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:27.586366 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:27.620805 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:27.624702 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:27.627343 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:27.633777 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:27.663251 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:27.819303 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:27.856267 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling FlushMRSOp(c0e88937e1ec49bfbaffe535e445e40a): perf score=0.033335
W20240501 20:14:27.893258 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:14:27.897130 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:27.911819 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:14:27.917948 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:27.925506 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:27.929033 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:27.959339 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:27.980995 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:27.986693 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: FlushMRSOp(c0e88937e1ec49bfbaffe535e445e40a) complete. Timing: real 0.130s user 0.038s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":372,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1659,"drs_written":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":992,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11156,"threads_started":2,"wal-append.queue_time_us":10790}
I20240501 20:14:27.988304 29382 maintenance_manager.cc:419] P 212de34e3aa24fa1be12c7421df985fc: Scheduling UndoDeltaBlockGCOp(c0e88937e1ec49bfbaffe535e445e40a): 4581 bytes on disk
I20240501 20:14:27.989774 29273 maintenance_manager.cc:643] P 212de34e3aa24fa1be12c7421df985fc: UndoDeltaBlockGCOp(c0e88937e1ec49bfbaffe535e445e40a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":144,"lbm_reads_lt_1ms":4}
W20240501 20:14:27.994848 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:14:28.003119 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:28.006805 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:28.048741 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:28.177196 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:28.180375 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:28.185489 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:46293, user_credentials={real_user=slave}} blocked reactor thread for 55458.5us
I20240501 20:14:28.205576 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:28.270718 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:28.272392 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:28.274478 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:28.291724 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:28.329365 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:14:28.474855 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:28.494385 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:28.500471 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:28.564427 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:28.576222 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:28.607712 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:14:28.619493 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:28.619889 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:28.627923 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:28.770200 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:28.786070 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:28.793273 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:28.798413 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:28.843444 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:28.855553 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:28.912683 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:28.997076 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:14:29.004799 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:14:29.014768 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:29.028455 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:29.028447 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:29.030202 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:29.037752 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:29.083936 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:29.090901 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:29.106369 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:29.119647 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:29.184629 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:29.197417 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:14:29.331893 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:29.338145 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:29.341288 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:29.342042 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:29.351022 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:29.376984 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:29.387604 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:29.404945 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:29.409961 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:29.422199 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:29.489862 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:14:29.706832 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:29.716455 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:29.729382 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:29.739100 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:29.769964 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:29.807075 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:29.833359 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:29.848395 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:29.889039 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:29.916677 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:29.928517 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:29.947096 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:29.969084 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:29.972065 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:29.988515 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:29.996434 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:30.017246 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:30.055328 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:30.159360 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:30.182888 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:30.199487 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:30.210697 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:30.237200 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:30.246052 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:14:30.300413 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:30.304167 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:30.315855 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:30.348873 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:14:30.385398 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:14:30.456252 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:30.486819 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:30.516520 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:30.539186 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:30.567353 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:30.644996 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:30.677714 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:30.698953 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:30.711661 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 63207.9us
W20240501 20:14:30.845399 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:30.847422 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:30.854022 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:30.857371 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:30.868490 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:30.944788 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:30.958849 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:30.981484 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:31.232080 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:31.248792 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:31.282186 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:31.306035 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:31.309074 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:31.312135 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:31.327560 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:31.338200 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:31.365165 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:31.367640 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:31.373095 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:31.389600 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:31.423271 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:31.425135 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:31.455090 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:31.501895 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:31.624179 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:14:31.686381 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:14:31.723812 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:14:31.793591 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:31.813650 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:31.821574 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:31.840474 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:31.912492 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:31.951614 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:31.959259 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:31.974038 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:32.023375 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:32.025929 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:32.060286 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:32.064770 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:32.106348 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:32.120347 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:32.138129 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:32.140375 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:32.155333 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:32.184022 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:32.221587 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:32.266201 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:32.323467 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:32.326261 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:32.374495 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:32.380563 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:32.385704 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:32.405261 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:32.423327 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:32.437331 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:32.448823 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:32.494273 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:32.668911 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:32.683002 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:32.692304 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:32.725322 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:32.730310 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:32.740546 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:14:32.740669 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:32.747932 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:32.748284 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:14:32.757158 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:32.829020 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:14:32.857230 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:32.917964 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:32.920917 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:32.938853 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:32.952126 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:33.011094 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:33.017593 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:33.032085 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:33.133704 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:14:33.174741 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:33.204190 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:33.232805 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:33.262825 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:33.269132 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:33.293745 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:33.299636 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:33.326751 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:33.486129 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 20:14:33.509485 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling FlushMRSOp(f034a679870b44caa238f04a8b49d302): perf score=0.033367
I20240501 20:14:33.594234 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:33.597457 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:33.640089 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:33.671564 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:33.693159 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:33.719378 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:33.748286 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:33.809211 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:33.877285 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: FlushMRSOp(f034a679870b44caa238f04a8b49d302) complete. Timing: real 0.367s user 0.051s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":353,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":786,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19150,"threads_started":2,"wal-append.queue_time_us":14606}
I20240501 20:14:33.883399 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(f034a679870b44caa238f04a8b49d302): 4581 bytes on disk
I20240501 20:14:33.884083 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(f034a679870b44caa238f04a8b49d302) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 20:14:33.911268 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:33.919759 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:14:33.960788 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:33.978271 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:33.992689 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:34.004976 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:34.005134 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:34.010900 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:34.017498 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:14:34.073592 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:34.123066 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:34.228736 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:34.232103 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:34.257591 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:34.273176 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:34.274933 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:34.283648 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:34.295305 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:34.329878 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:34.339339 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:34.402297 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:14:34.414669 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:14:34.425892 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:34.433048 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:34.448779 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:34.454676 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:34.465878 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:34.481971 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:34.526340 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:34.595710 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:34.701905 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:34.710209 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:34.731248 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:14:34.744900 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:14:34.745263 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:34.756820 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:34.759976 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:34.804886 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:34.806254 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:34.807660 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:34.809640 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:34.843168 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:34.915050 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:14:34.959336 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:14:35.117429 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:35.118199 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:35.120047 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:35.160454 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:35.195487 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:35.229211 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:35.261416 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:14:35.269553 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:35.287735 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:35.298537 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:35.376660 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:35.403625 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:35.406888 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:35.435051 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:35.438323 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:35.452811 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:35.460716 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:35.466104 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:35.635541 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:35.645027 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:35.676935 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:35.684610 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:35.693475 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:35.706478 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:35.718283 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:35.729698 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:35.743144 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:35.782274 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:35.790058 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:35.852643 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:35.863492 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:35.902050 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:35.919656 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:36.028026 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:36.132395 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:36.156347 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:36.210299 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:36.219084 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:36.238026 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:36.240198 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:36.272042 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:36.400301 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:36.434866 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:36.439565 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:36.473022 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:36.491981 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:36.492827 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:36.511945 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:36.517796 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:36.559211 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:36.758435 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:36.834342 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:36.842526 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:36.848102 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:36.883219 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:36.898514 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling FlushMRSOp(f034a679870b44caa238f04a8b49d302): perf score=0.033388
W20240501 20:14:36.908304 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:14:36.908979 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:36.937608 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:36.952188 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:36.960418 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:36.971320 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:36.990237 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:36.992429 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:36.997435 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:37.001338 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:37.028731 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:37.032629 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:37.084224 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:37.098080 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:37.111187 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:37.112268 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:37.195329 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:37.197643 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:37.200296 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:37.243292 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:37.298175 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:37.308494 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:37.326013 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:37.327452 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:14:37.335222 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:14:37.399426 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:14:37.420292 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:14:37.429212 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: FlushMRSOp(f034a679870b44caa238f04a8b49d302) complete. Timing: real 0.530s user 0.055s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10371,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1639,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1055,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10662,"threads_started":2,"wal-append.queue_time_us":476}
I20240501 20:14:37.431226 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(f034a679870b44caa238f04a8b49d302): 4581 bytes on disk
I20240501 20:14:37.446035 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(f034a679870b44caa238f04a8b49d302) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240501 20:14:37.525027 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:37.527318 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:37.532676 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:37.544288 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:37.548646 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:37.567670 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:37.606611 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:37.627580 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:37.638952 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:37.678004 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:37.724380 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:37.894431 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:37.930320 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:37.937618 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:37.960088 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:37.965873 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:37.971591 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:37.972685 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:37.981312 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:38.136531 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:14:38.277264 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:38.297691 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:38.340859 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:38.347894 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:38.358330 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:38.360659 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:38.382263 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:38.400475 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:38.462221 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:38.465230 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:38.483644 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:38.490473 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:38.511864 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 20:14:38.525242 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 66168.9us
I20240501 20:14:38.588462 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:38.590754 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:38.635529 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:38.858299 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:38.865526 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:38.874013 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:38.899518 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:38.922724 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:38.952261 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:38.977702 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:14:38.985350 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:38.986135 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:39.013861 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:39.136618 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:14:39.204967 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:14:39.218883 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 52034.2us
W20240501 20:14:39.276276 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:39.326818 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:39.340939 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:39.361471 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:39.363000 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:39.367419 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:39.374557 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:39.431353 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:39.435961 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:39.438951 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:39.454509 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:39.480551 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:39.480930 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:39.483724 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:39.503726 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:39.542330 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:39.593034 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:39.632300 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:39.646160 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:39.713017 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:39.713924 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:14:39.723392 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:39.742756 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:39.767326 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:39.772614 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:39.779882 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:14:39.790409 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:39.809057 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:14:39.833101 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:14:39.884918 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:14:39.910573 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:39.959054 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:39.994113 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:39.994813 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:40.008193 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:40.027001 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:40.029559 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:40.031878 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:40.044508 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:40.063998 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:40.230247 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:40.275027 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:40.281965 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:40.312909 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:40.339541 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:40.343643 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:40.364349 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:40.377296 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:40.405184 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:40.600811 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:40.618002 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:40.626519 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:40.695051 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:40.706981 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:40.744011 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:40.749373 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:40.760303 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:40.780478 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:40.787249 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:40.802984 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:40.806962 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:40.827109 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:40.847512 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:40.883980 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:40.944087 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:41.046064 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 86061.1us
W20240501 20:14:41.081892 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:41.153859 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:41.196110 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:41.206465 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:41.211612 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:41.225068 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:41.249357 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:41.309497 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:41.455194 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:41.490013 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:41.502483 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:41.503760 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:41.531205 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:41.534888 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:41.585654 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:41.616756 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:41.631600 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:41.673755 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:41.875286 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:41.908833 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:41.922607 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:41.950305 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:41.954167 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:41.956447 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:42.006824 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:42.007756 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:42.021731 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:42.029521 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:42.041612 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:42.041662 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:42.045940 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:42.108464 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:14:42.113176 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:42.117092 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:42.119041 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:42.167410 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:14:42.189816 27330 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:46843, user_credentials={real_user=slave}} blocked reactor thread for 99174.6us
W20240501 20:14:42.192085 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:14:42.198925 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:14:42.209843 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:14:42.217370 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:14:42.298130 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:42.366029 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:42.446033 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:42.460206 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:42.468374 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:42.486482 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:42.523917 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:42.561898 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:42.564355 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:42.568245 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:42.610252 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:14:42.699316 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:42.848671 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:42.877426 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:42.902859 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:42.912897 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:42.918751 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:42.952229 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:42.957728 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:42.985728 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:43.004914 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:43.182345 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:14:43.249629 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:43.257964 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:43.280263 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:43.302770 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:43.304318 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:43.323825 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:43.326212 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:43.354079 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:43.378360 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:43.383198 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:43.450192 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:43.453708 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:43.481961 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:43.543972 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:43.583067 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:43.602455 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:43.683706 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:43.703691 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:43.721937 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:14:43.724416 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:43.757061 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:43.808588 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:43.815557 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:43.816428 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:43.860266 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:43.889600 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:14:44.009680 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:44.015017 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:44.031253 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:44.050295 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:44.061285 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:44.127013 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:44.134397 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:14:44.135447 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:44.152544 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:44.178947 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 20:14:44.235325 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:44.348496 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:14:44.423096 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:44.441462 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:44.462327 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:44.478377 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:44.543468 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:44.574592 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:44.575829 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:44.637689 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:14:44.667850 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:14:44.684037 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:44.736716 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:44.753582 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:14:44.756723 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:44.763042 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:44.779613 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:14:44.782531 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:44.785236 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:44.786942 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:44.800784 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:44.827162 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:44.842475 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:44.872081 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:14:45.018443 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:45.039463 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:45.043542 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:45.069892 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:45.078537 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:45.082746 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:14:45.090209 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:14:45.091277 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:45.110891 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:45.118541 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:14:45.344884 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:45.377112 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:45.386499 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:45.417624 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:45.438760 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:45.441056 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:45.446764 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:14:45.462978 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:45.495160 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:45.502350 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:45.515527 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:45.539640 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:45.556416 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:45.602106 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:45.608471 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:45.612411 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:45.836691 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:45.838508 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:45.849257 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:45.872025 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:45.899497 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:45.921766 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:45.928592 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:46.002185 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:46.035260 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:14:46.115655 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:14:46.185654 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:46.233743 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:46.280570 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:46.365566 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:46.371165 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:46.391595 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:46.410307 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:46.614403 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:46.654613 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:46.671135 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:46.688578 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:46.690212 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:46.708232 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:14:46.710475 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:46.712966 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:46.741601 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:46.746912 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:46.747750 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:46.818480 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:46.825465 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:46.860036 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:46.881932 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:46.890564 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:46.907212 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 60390.7us
I20240501 20:14:46.937726 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:46.969166 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:46.974733 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:14:46.983155 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:14:46.984705 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:46.988049 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:47.012828 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:47.030733 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:47.051882 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:14:47.053517 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:47.086685 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:14:47.139232 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:14:47.149073 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:47.190461 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:14:47.217434 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:14:47.346851 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:47.363927 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:47.369961 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:47.381976 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:47.417009 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:47.443775 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:47.447126 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:47.475205 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:14:47.518318 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:14:47.529191 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:14:47.551470 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:14:47.565819 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:47.596319 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:47.612314 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:47.650570 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:47.666018 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:47.668068 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:47.678236 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:47.705889 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:47.765540 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:14:47.996062 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:48.014384 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:48.035075 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:48.086330 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:48.133584 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:48.142410 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:48.176247 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:48.225385 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:48.368686 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:48.374455 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:48.391485 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:48.399089 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:48.425971 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:48.436059 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:48.457299 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:48.466953 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:14:48.474740 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:14:48.490074 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:14:48.524647 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:48.528721 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:48.607091 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:48.610108 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:48.618108 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:48.628381 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:48.630795 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:48.649483 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:48.726970 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:14:48.745566 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:48.795297 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:48.800369 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:48.817615 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:48.922279 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:48.958917 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:48.967111 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:49.110647 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:49.135921 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:49.137449 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:49.138648 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:14:49.138857 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:14:49.194099 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:49.212307 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:49.213404 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:49.253183 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:49.363991 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:14:49.372131 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:14:49.399941 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:49.415907 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:49.417860 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:49.429220 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:49.444903 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:49.473373 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:14:49.489923 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:14:49.509194 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:49.517817 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:49.547961 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:49.604378 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:14:49.634085 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:14:49.729708 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:14:49.780035 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:14:49.798743 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:49.817476 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:14:49.831377 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:49.897501 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 63355.5us
I20240501 20:14:49.924711 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:49.935740 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:49.968482 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:14:50.002419 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:14:50.020282 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:50.022109 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:50.047732 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:50.047825 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:50.050261 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:50.073213 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:50.093902 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:50.097359 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:50.125499 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:50.149135 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:50.187462 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:50.203402 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:14:50.208292 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:14:50.377743 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:50.436158 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:50.438948 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:50.452025 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:50.462661 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:50.507977 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:50.540868 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:50.542783 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:50.635087 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:50.674117 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:50.680820 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:50.705938 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:50.744386 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:50.764067 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:50.791982 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:50.908588 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:50.952455 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:50.981726 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:51.012279 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:51.013916 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:51.046373 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:14:51.071060 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:51.133235 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:51.133630 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:51.207192 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:51.324540 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:51.355593 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:51.364933 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:51.371522 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:51.421962 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:51.422384 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:51.447265 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:51.476994 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:51.571882 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:51.578379 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:51.593773 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:51.635458 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:51.662101 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:51.713907 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:51.715807 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:51.737061 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:51.879052 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:51.905095 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:14:51.908097 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:51.928622 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:51.934377 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:51.948762 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:51.953114 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:51.965462 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:51.975369 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:51.983104 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:52.013425 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:52.023859 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:52.073689 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:14:52.083794 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:14:52.085242 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:52.095486 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:52.095638 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:52.116314 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:52.124979 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:14:52.154217 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:52.171456 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:52.234562 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:52.266813 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:14:52.380473 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:52.402892 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:52.405571 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:52.432011 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:52.464213 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:52.467849 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:52.552525 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:52.555512 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:52.604956 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:14:52.635957 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:52.652245 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:14:52.662294 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:14:52.686230 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:52.694658 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:52.726192 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:52.737538 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:52.741933 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:52.773988 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:52.826944 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:53.136279 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:53.175376 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:53.280014 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:53.303735 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:53.309753 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:53.383185 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:53.401847 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:53.412747 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:53.412786 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:53.414330 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:53.487365 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:53.533421 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:53.542098 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:53.566721 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:14:53.575523 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:53.581724 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:53.654482 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:53.677565 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 75142.7us
W20240501 20:14:53.694753 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:53.709914 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:53.795264 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:14:53.851564 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:53.882834 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:53.891429 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:53.920770 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:53.962720 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:53.968164 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:53.973397 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 152787us
I20240501 20:14:53.980522 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:53.990270 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:53.993499 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:53.993521 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:54.003844 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:54.040022 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:54.048780 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:54.052631 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:54.057380 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:54.186249 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:54.301712 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:54.365249 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:54.376250 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:54.389004 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:54.392516 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:54.406672 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:14:54.420185 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:54.426714 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:54.445201 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:14:54.458374 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:54.484910 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:14:54.502447 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:54.522441 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:14:54.535609 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:54.598637 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:54.599689 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:54.605340 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:54.624508 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:54.692128 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:54.696483 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:54.712074 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:54.735428 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:14:54.766094 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:54.781554 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:14:55.009608 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:55.053292 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:55.064549 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:55.068517 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:55.079946 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:55.097247 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:55.106834 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:55.114841 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:55.148448 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:55.152096 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:55.165047 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:14:55.166231 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:55.167047 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:55.191578 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:55.216166 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:55.259382 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:55.267127 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:55.283465 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:55.287358 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:55.310806 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:14:55.336028 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:55.339954 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:55.383036 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:55.398833 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:55.407411 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:55.474447 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:55.482193 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:55.758342 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:55.772305 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:55.803220 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:55.811764 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:55.823000 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:55.837150 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:55.879827 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:55.889645 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:55.900161 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:56.072059 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:56.093520 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:56.108392 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:56.125989 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:56.132176 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:56.145967 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:56.184886 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:56.191725 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:56.281282 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:56.421587 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:56.435590 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:56.453166 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:56.486737 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:56.495244 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:56.530941 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:56.535892 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:56.604202 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:56.638864 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:56.688738 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:14:56.712625 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:56.717769 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:56.730053 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:56.752313 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:56.754521 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:56.763720 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:56.778867 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:56.812186 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:56.952903 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:14:56.991230 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:57.007841 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:57.028904 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:57.049934 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:57.056083 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:57.103778 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:57.130015 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:57.132282 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:57.203734 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:57.209295 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:57.239873 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:57.271343 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:57.349472 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:57.380023 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:57.383749 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:57.391809 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:57.429493 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:57.457908 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:57.484474 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:57.495967 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:14:57.517952 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:14:57.579092 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:14:57.610929 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:14:57.632800 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:57.636648 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:57.656862 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:57.678042 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:57.693121 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:57.777199 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:57.778728 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:57.806303 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:57.925818 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:14:57.942066 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:57.967460 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:57.993594 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:58.007354 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:58.009130 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:58.038898 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:58.063529 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:58.244716 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:58.289184 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:58.305020 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:58.307926 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:58.308431 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:58.339697 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:58.340227 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:58.356045 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:58.391216 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:58.534914 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:58.541898 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:58.567710 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:58.574146 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 20:14:58.577226 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:58.586735 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:58.599280 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:58.623109 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:14:58.631462 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:58.668789 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:58.690538 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:58.742010 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:58.751240 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:58.788241 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:58.791070 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:58.818933 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:14:58.823590 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:14:58.900746 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:14:58.913589 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:59.083865 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:59.100615 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:59.146461 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:14:59.178263 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:14:59.191907 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:14:59.193719 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:59.213181 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:59.251487 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:59.260936 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:59.270170 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:14:59.281193 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:59.362695 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:59.375883 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:14:59.379973 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:59.394682 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:59.406919 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:59.445603 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:14:59.455847 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:14:59.465586 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:14:59.482208 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:59.502887 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:14:59.597249 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:14:59.666260 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:14:59.790199 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:14:59.844954 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:59.866387 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:14:59.876119 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:14:59.899967 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:14:59.900295 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:14:59.922416 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:14:59.922698 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:14:59.957617 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:14:59.963300 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:14:59.991309 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:00.023530 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:00.064565 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:15:00.096300 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:15:00.107609 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:15:00.121042 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:00.129374 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:00.156873 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:00.161078 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:15:00.184232 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:00.206422 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:00.210598 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:00.253376 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:00.290714 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:00.301767 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:00.347856 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:00.393945 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:00.397576 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:00.455293 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:00.613337 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:00.638216 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:00.662566 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:00.678706 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:00.679030 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:00.718935 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:00.720100 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:00.734177 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:00.754225 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:15:00.863734 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:00.882428 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:00.893397 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:00.916445 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:00.953696 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:00.997552 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:01.008821 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:01.009739 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:01.147471 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:01.364629 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:01.412434 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:01.417953 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:01.453557 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:01.458675 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:01.491523 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:01.494542 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:01.496759 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:01.525957 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:01.530889 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:01.543035 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:01.550388 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:15:01.611228 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:01.632292 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:01.643970 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:01.685839 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:01.729105 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:01.779314 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:01.786154 27068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 68413.6us
I20240501 20:15:01.818915 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:01.826269 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:01.833487 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:01.855885 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:01.898963 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:01.907269 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:01.928349 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:02.100673 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:15:02.132261 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:02.134361 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:02.150097 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:02.169162 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:02.217624 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:02.238821 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:02.274354 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:02.312300 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:02.330502 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:15:02.330775 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:02.330897 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:15:02.399793 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:15:02.468235 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:15:02.548593 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:15:02.591804 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:02.602380 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:02.617038 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:02.621870 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:02.698971 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:02.699787 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:02.706359 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:02.750834 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:02.758105 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:02.806466 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:15:02.864759 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:02.921811 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:02.931519 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:02.942960 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:02.963840 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:02.984784 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:03.001740 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:03.028012 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:03.047570 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:03.083158 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:03.092275 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:03.162910 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:03.175609 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:03.198443 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:03.216876 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:03.229130 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:03.248883 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:03.327266 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:03.461481 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:03.462703 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:03.480607 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:03.498296 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:03.509629 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:03.510129 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:03.510891 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:03.544829 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:03.617009 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:15:03.707280 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:03.738500 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:03.762873 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:03.767973 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:03.803587 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:03.820353 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:03.857868 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:03.872972 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:15:03.926803 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:03.930542 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:04.079183 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:15:04.080140 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:04.319708 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:04.334811 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:04.367026 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:04.367159 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:04.386849 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 60659.5us
I20240501 20:15:04.430918 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:04.476600 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 85256.4us
I20240501 20:15:04.499439 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:04.508200 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 127739us
I20240501 20:15:04.509338 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:04.511781 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:04.537467 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:04.537833 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:04.547945 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:04.593866 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:15:04.595964 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:04.606974 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:04.627319 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:04.627754 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:04.643354 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:04.648195 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:04.654803 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:04.724397 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:04.729949 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:04.735632 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:15:04.746883 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:04.757313 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:04.767357 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:04.773599 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:15:04.783867 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:04.809007 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:15:04.877580 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:15:04.970073 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:05.012336 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:05.020220 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:05.022701 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:05.024467 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:05.043182 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:05.061998 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:05.083418 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:05.101305 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:05.169842 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:05.181059 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:15:05.368580 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:05.434230 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:05.437546 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:05.448048 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:05.477661 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:05.509323 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:15:05.524422 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:05.547209 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:05.560226 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:05.592646 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:05.676023 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:05.726958 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:05.780462 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:05.806703 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:05.816200 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:05.835706 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:05.839013 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:05.859265 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:05.878098 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:05.890172 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:05.913192 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:05.942608 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:05.952646 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:05.956538 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:15:05.994740 27200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:43861, user_credentials={real_user=slave}} blocked reactor thread for 64061.2us
I20240501 20:15:06.004349 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:06.015573 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:06.024693 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:06.158252 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:15:06.197945 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 55179.2us
W20240501 20:15:06.335482 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:06.376152 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:06.377375 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:06.418457 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:06.434304 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:06.481545 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:06.482161 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:06.487586 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:06.497105 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:06.572094 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:06.687373 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:06.989339 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:07.010262 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:07.011412 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:07.016250 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:07.023968 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:07.030154 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:07.031999 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:07.046458 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:07.051178 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:07.116472 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:07.116644 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:07.138931 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:07.177208 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:07.181421 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:07.200510 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:07.217660 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:07.234669 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:07.235949 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:07.237763 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:07.255219 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:07.274817 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:07.307579 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:07.327203 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:07.354367 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:07.382652 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:07.414733 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:07.524693 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:07.574446 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:07.581468 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:07.589375 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:15:07.619966 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:15:07.632560 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:07.638314 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:07.670169 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:07.694411 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:07.726495 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:07.729106 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:07.858062 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:07.901222 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:07.919571 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:07.922983 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:07.924036 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:07.989612 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:08.057467 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:08.068084 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:08.080808 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:15:08.103758 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:08.205579 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:08.269042 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:08.283365 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:08.293036 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:08.309011 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:08.326476 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:08.338798 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:08.341754 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:08.377763 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:08.576133 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:08.589464 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:08.594800 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:08.665316 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:08.669807 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:15:08.670847 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:08.681341 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:08.707861 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:08.708284 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:08.710534 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:08.726573 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:08.745067 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:08.748865 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:08.774135 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:08.783336 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:08.817325 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:08.846134 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:08.904381 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:15:08.964924 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:15:08.966012 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:09.049080 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:09.083101 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:09.149708 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:09.160552 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:09.167450 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:09.191779 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:09.206588 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:09.206918 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:09.307762 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 144990us
W20240501 20:15:09.421694 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:15:09.467185 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:09.500308 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:09.528332 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:09.530333 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:09.560205 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:09.570778 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:09.584300 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:09.597194 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:09.601137 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:09.618760 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:09.631407 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:09.643203 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:09.648516 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:09.716496 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:09.725950 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:09.745532 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:09.762250 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:09.820919 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:15:09.823308 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:15:09.906090 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:15:09.935518 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 20:15:09.987017 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:09.991271 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:09.993161 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:09.999989 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:10.004546 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:10.026219 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:10.043608 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:10.072413 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:10.234771 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:10.265132 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:15:10.319969 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:10.341377 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:10.348245 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:10.360859 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:10.382565 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:10.412688 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:10.469216 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:10.469767 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:10.575098 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:15:10.576478 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:15:10.705374 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:15:10.784808 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:10.802251 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:10.819052 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:10.853130 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:10.923084 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:10.931416 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:10.971014 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:10.978561 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:10.991370 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:11.004076 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:11.004601 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:11.010744 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:11.036538 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:11.085557 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:11.086248 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:11.087373 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:11.145962 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:15:11.353202 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:11.358986 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:11.371580 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:11.380859 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:11.383796 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:11.406138 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:11.423074 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:11.431968 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:11.464932 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:11.600235 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:11.638031 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:11.639335 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:11.651012 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:11.670445 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:11.671259 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:11.696760 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:11.732581 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:11.845500 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:11.884618 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:11.903283 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:11.908058 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:11.912397 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:11.923602 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:11.936344 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:11.969648 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:11.985229 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:11.986302 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:12.164727 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:15:12.174573 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:12.252689 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:12.272866 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:12.301658 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:12.317824 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:12.317974 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:12.327780 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:12.345422 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:12.395388 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:12.491366 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:12.592424 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:12.595711 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:12.604703 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:12.642724 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:12.649082 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:12.649960 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:12.693964 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:12.714823 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:12.756662 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:12.761546 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:12.807510 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:12.817360 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:15:12.820811 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:12.826030 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:12.834380 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:12.834815 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:12.858319 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:12.960116 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:15:12.987763 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:12.997097 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:12.997593 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:13.032402 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:13.060319 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:15:13.071127 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:13.116451 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:13.129084 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:13.131089 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 48710.5us
I20240501 20:15:13.143095 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:13.260967 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:15:13.284397 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:15:13.341734 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:13.376536 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:13.377848 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:13.392650 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:13.416306 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:13.447568 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:13.455963 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:13.474809 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:13.623339 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:13.823377 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:13.825592 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:13.831120 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:13.839212 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:13.845276 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:13.857803 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:15:13.862413 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:13.913262 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:13.924288 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:13.925706 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:13.943902 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:13.978809 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 20:15:13.984002 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:14.028604 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:14.045717 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:14.052042 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:14.079855 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:14.090937 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:14.091660 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:14.125635 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:14.127683 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:14.218838 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:14.224470 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:14.250586 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:14.252434 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:14.269110 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:14.378367 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:14.451856 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:14.522600 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:14.545152 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:14.556619 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:14.570217 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:14.579759 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:14.600173 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:14.612215 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:14.616375 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:14.643644 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:14.659863 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:14.687224 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:14.696270 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:14.699605 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:15:14.700202 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:14.737286 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:14.820204 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:14.840317 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:14.874542 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:15:14.899418 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:14.970705 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:15:15.015311 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:15:15.073612 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:15:15.153749 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:15.191524 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:15.235981 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:15.262755 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:15.267397 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:15.268718 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:15.271651 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:15.319047 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:15.421276 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:15.448788 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:15.475811 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:15.484453 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:15.491142 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:15:15.498277 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:15:15.530974 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:15.564615 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:15.571616 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:15.578764 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:15.595193 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:15.697451 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:15:15.722056 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:15.825762 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:15.828644 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:15.844247 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:15.848358 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:15.902586 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:15.940332 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 91413.2us
W20240501 20:15:15.967499 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:15.977932 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:15.979964 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:15.985992 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:16.190647 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 46321.5us
I20240501 20:15:16.193732 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:16.217967 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:16.272183 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:16.291574 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:16.327622 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:16.330055 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:16.355269 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:16.406270 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:16.434405 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:16.435094 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:16.460281 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:16.460669 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:16.479072 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:16.489822 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:16.494382 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:16.542688 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:16.660154 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:16.678495 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:16.754777 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:16.760757 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:16.777510 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:16.791730 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:16.819336 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:16.839603 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:16.855410 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:16.979846 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:16.989285 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:17.002785 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:17.009999 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:17.024446 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:17.030243 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:17.037338 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:17.041005 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:17.051183 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:17.112757 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:15:17.233003 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:15:17.298853 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:15:17.423138 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:17.452332 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:17.458170 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:17.506311 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:17.516043 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:17.534670 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:17.567992 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:17.671998 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:17.727988 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:15:17.786581 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:17.814232 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:17.832808 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:17.841812 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:17.842060 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:17.853809 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:17.879902 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:17.879968 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:17.883558 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:17.941717 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:17.942869 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:17.953606 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:17.968416 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:17.975478 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:17.991910 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:17.999724 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:18.025609 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:18.091559 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:18.120762 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:15:18.180656 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:15:18.250571 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:18.260852 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:18.276074 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:18.301330 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:18.339445 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:18.347512 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:18.355628 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:18.357919 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:18.451763 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:18.584748 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:18.600981 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:18.637421 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:18.645110 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:18.647325 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:18.743522 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:18.753527 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:18.796388 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:18.856735 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:18.910497 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:18.920325 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:18.957155 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:18.960361 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:18.979452 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:15:19.024008 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 20:15:19.025614 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:19.028775 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:19.055018 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:19.071020 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:19.097630 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:15:19.194648 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:15:19.241439 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:19.290144 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:19.306715 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:19.309037 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:19.326725 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:19.379559 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:19.387991 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:19.393019 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:19.505321 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:19.511010 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:19.520184 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:19.531773 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:19.564648 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:19.576123 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:19.578933 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:19.587177 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:19.634606 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:19.708588 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:15:19.720952 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:19.860404 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:19.869467 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:19.875763 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:19.895437 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:19.899261 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:19.955569 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:19.972394 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:19.978744 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:20.015970 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:20.028569 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:20.056738 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:20.080729 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:15:20.082759 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:15:20.083886 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:20.098747 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:20.139217 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:15:20.152392 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:20.154110 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:20.154817 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:20.213624 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:20.315207 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:20.453205 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:20.473865 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:20.487650 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:20.489094 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:20.501616 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:20.517922 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:20.548916 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:20.556257 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:20.567719 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:20.576303 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:20.591349 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:20.617982 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:20.635788 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:20.639545 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:20.662331 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:15:20.705202 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:20.722568 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:20.763010 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:20.774988 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:20.783684 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:20.837512 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:20.870771 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:20.871351 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:20.893405 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:20.910742 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:20.956744 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:15:21.065222 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:21.084563 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:21.139838 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:21.185905 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:21.186925 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:21.224300 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:21.260787 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:21.262280 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:21.286077 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:21.380297 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:21.482203 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:21.484915 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:21.485179 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:21.499603 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:21.512099 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:21.558357 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:21.590004 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:21.590081 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:21.666749 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:15:21.769301 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:15:21.798538 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:21.855135 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:21.855310 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:21.903234 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:21.910251 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:21.913013 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:21.928442 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:21.944828 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:22.074939 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:22.083771 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:22.144665 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:22.153303 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:22.224617 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:22.226882 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:22.232092 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:22.244863 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:22.270133 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:22.282068 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:22.297902 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:22.413070 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:22.420773 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:22.440080 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:22.465982 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:22.496681 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:22.506682 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:22.544021 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:22.607942 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:15:22.625159 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:15:22.640508 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:22.658926 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:22.677394 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:22.685806 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:22.700392 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:22.720511 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:22.770489 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:22.797899 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:22.808720 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:22.943334 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:22.957460 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:22.973870 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:22.982163 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:23.002357 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:23.021330 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:23.043552 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:23.068264 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:23.113067 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:23.255545 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:23.344650 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:15:23.371466 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:23.401896 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:23.412631 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:23.439033 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:23.445271 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:23.458194 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:23.460837 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:23.473829 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:23.473940 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:23.485927 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:23.511178 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:23.521919 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:23.617130 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:23.618304 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:23.637590 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:23.706246 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:23.835918 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:15:23.866492 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:23.929378 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:23.932658 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:23.956357 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:23.984318 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:23.995087 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:15:23.995378 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:23.996029 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:24.008576 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:24.024384 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:24.208678 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 20:15:24.243110 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:15:24.290839 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:24.339221 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:24.343425 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:24.356788 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:24.389369 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 48481us
I20240501 20:15:24.425853 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:24.426853 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:24.459223 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:24.469246 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:24.486943 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:24.651281 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:24.665835 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:24.668040 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:24.690218 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:24.695871 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:24.699576 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:24.702865 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:24.712735 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:24.727216 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:24.839373 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:24.847343 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:24.857439 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:24.864567 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:24.873587 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:24.917914 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:24.924286 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:24.959375 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:24.962385 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:25.029042 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:25.057616 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:25.077239 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:25.098070 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:25.125866 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:25.140192 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:15:25.141108 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:15:25.159852 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:25.178835 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:25.179951 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:25.182977 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:15:25.214536 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:15:25.286537 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:25.426614 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:25.427013 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:25.427465 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:25.442857 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:25.476461 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:25.482124 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:25.504107 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:25.541383 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:25.674798 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:25.801522 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:25.829828 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:15:25.831336 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:25.839165 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:25.845304 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:25.887723 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:25.933413 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:25.958921 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:25.967173 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:25.972782 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:26.016487 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:26.042699 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:26.049928 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:26.065551 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:26.068835 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:26.069000 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:26.080161 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:26.141074 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:26.174005 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:26.180665 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:26.206789 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:26.229184 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:26.231595 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:26.243968 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:26.254050 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:26.320178 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:15:26.398734 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:15:26.472662 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:26.483726 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:26.490557 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:15:26.498032 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:26.506568 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:26.510964 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:26.514971 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:26.529219 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:26.543994 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:27.094586 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:27.098147 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:15:27.107259 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:15:27.130553 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:27.147487 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:27.249305 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:27.288175 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:27.306196 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:27.316251 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:27.326840 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:27.346750 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:27.378041 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:27.413797 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:27.424809 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:27.437216 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:27.442683 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:15:27.449978 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:27.471127 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:27.473825 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:27.486948 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:27.525946 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:27.529098 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:27.532780 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:27.533946 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:27.538048 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:27.540481 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:27.556010 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:27.583061 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:27.583537 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:27.591379 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:27.618297 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:27.618360 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:27.647231 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:27.664093 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:27.723572 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:15:27.739184 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:15:27.860775 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:15:27.907696 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:15:27.982316 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:27.983717 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:27.996280 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:28.004635 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:28.011389 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:28.035281 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:28.053151 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:28.160210 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:28.211845 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:15:28.273803 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:15:28.318938 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:15:28.454315 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:28.467053 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:28.489245 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:28.490113 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:28.560333 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:28.566295 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:28.639451 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:28.744829 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:28.775091 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:28.776022 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:28.791393 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:28.792987 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:15:28.813942 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:28.835173 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:28.869599 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:28.883574 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:28.886749 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:28.891724 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:28.951965 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:28.979589 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:29.003999 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:29.018805 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:29.023337 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:29.039189 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:29.043399 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:29.093653 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:29.139204 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:29.160377 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:29.176431 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:29.186921 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:29.210098 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 20:15:29.222144 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 20:15:29.223040 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:29.239104 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:29.264995 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:29.316370 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:29.520480 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:29.534070 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:29.537834 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:29.562885 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:29.606400 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:29.607229 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:29.630790 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:29.635862 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:29.771131 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:29.787566 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:15:29.817716 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:29.820762 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:29.842672 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:29.848567 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:15:29.851140 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:29.857342 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:29.904675 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:29.935767 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:29.936466 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:30.085497 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 49100us
I20240501 20:15:30.114024 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:30.146883 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:30.180064 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:30.231981 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:30.249792 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:30.261320 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:30.275050 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:15:30.277866 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:30.307206 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:30.347415 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:15:30.358986 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:15:30.415781 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:30.463587 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:30.469621 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:15:30.479264 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:30.480859 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:30.504263 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:30.535864 27068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 74788us
I20240501 20:15:30.542670 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:30.555613 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:30.562361 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:30.583120 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:30.589365 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:30.589381 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:30.672398 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:15:30.793527 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:30.824476 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:30.828739 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:30.836226 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:30.847254 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:15:30.859671 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:30.937482 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:30.963433 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:31.101284 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:15:31.111102 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:31.111109 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:31.145251 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:31.158741 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:31.172559 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:31.203011 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:31.235484 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:31.254472 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:31.273836 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:31.500792 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:31.527226 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:31.575451 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:15:31.576452 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:31.581027 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:31.593122 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:31.618744 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:31.693049 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:31.695068 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:31.703931 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:31.715739 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:15:31.753458 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:15:31.766307 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:31.769232 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:31.799927 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:31.803596 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:31.808473 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:31.861569 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:31.933292 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:31.943892 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:15:31.961511 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:32.001765 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:32.035971 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:32.069895 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:32.091238 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:32.099572 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:32.298494 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:32.304337 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:32.314839 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:32.323046 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:32.355118 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:32.364903 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:32.398557 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:32.439405 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:32.440804 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:32.577261 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:32.598574 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:32.622431 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:32.640533 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:32.642585 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:32.670202 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:15:32.679303 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:15:32.687664 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:32.707059 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:32.761656 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:15:32.838318 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:15:32.871842 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:32.893327 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:32.908345 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:32.911028 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:32.923364 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:32.923815 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:15:32.943214 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:15:32.958318 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:33.004750 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:33.012780 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:33.193760 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:15:33.269182 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:15:33.288326 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:15:33.292899 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:33.359347 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:33.376199 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:33.385417 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:33.399014 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:33.403712 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:33.452546 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:33.468902 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:33.519413 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:33.524820 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:15:33.541937 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:33.590009 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:33.605767 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:33.628840 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:33.637756 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:33.662601 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:33.859365 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:33.970511 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:33.982964 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:34.007365 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:34.008749 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:34.050194 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:34.050364 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:34.080922 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:34.126819 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:34.133947 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:34.136557 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:34.155293 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:34.266017 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 20:15:34.273077 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:34.316202 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:15:34.327373 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:34.343439 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:34.368122 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:34.380260 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:34.388248 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:34.407606 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:34.418998 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:34.420513 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:34.425655 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:34.468888 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:34.508239 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:34.551295 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:34.602759 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:34.605324 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:34.684626 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 68991.3us
W20240501 20:15:34.792505 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:15:34.850524 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:34.850872 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:34.875505 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:34.885742 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:15:34.888023 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:34.924425 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:34.940423 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:34.941287 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:34.943903 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:34.984162 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:34.996795 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:34.997300 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:35.050278 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:35.095291 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:35.100600 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:35.107192 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:35.267460 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:15:35.318537 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:35.341318 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:15:35.411757 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:35.433439 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:35.440603 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:35.442869 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:35.465472 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:35.474001 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:15:35.481810 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:35.482820 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:35.506146 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:35.506811 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:35.509075 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:35.602380 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:15:35.654388 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 20:15:35.659191 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:35.684031 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:35.715205 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:35.720678 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:35.722689 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:35.754879 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:35.759665 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:35.826532 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:35.826879 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:35.841428 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:35.875792 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:35.895994 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:35.897274 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:35.909006 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:35.937135 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:15:35.939077 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:35.949599 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:15:35.961516 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:36.292791 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:36.296764 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:36.311905 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:36.315790 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:36.337239 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:36.347339 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:36.350876 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:36.377523 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:36.557936 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:36.559051 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:36.581334 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:15:36.608191 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:36.611912 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:36.646044 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:15:36.646679 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:36.699790 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:36.701121 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:36.726568 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:37.005455 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:15:37.058439 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:37.116644 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:37.121289 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:37.147400 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:15:37.162504 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:37.168289 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:37.251523 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:37.269675 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:37.269798 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:37.316175 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:37.318805 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:37.348206 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:37.362051 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:37.366243 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:37.380051 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:37.387151 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:37.451059 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:37.524199 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:37.526108 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:37.550065 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:37.555563 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:37.599887 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:37.604429 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:37.626669 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:37.652520 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:37.659667 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:15:37.690718 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:15:37.691516 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:15:37.797947 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:15:37.828682 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:37.863142 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:37.879143 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:37.887727 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:37.906152 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:15:37.909117 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:37.912127 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:37.925894 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:37.945145 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:38.196568 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:15:38.288726 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:38.357357 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:38.376037 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:15:38.429674 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:38.445892 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:38.454399 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:38.463845 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:38.485859 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:38.548331 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:15:38.549099 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:38.560542 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:38.871587 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:38.873982 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:38.897629 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:38.918581 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:38.927078 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:38.948871 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:38.955444 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:38.972435 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:38.972779 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:38.983655 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:39.032897 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:39.037024 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:39.048154 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:39.095758 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:39.132396 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:39.147871 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:39.159366 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:39.167809 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:39.198591 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:39.199813 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:39.246902 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:39.357136 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:15:39.360726 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:15:39.366432 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:39.377115 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:39.378866 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:39.474900 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:39.487941 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:39.498692 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:39.507443 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:39.513293 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:39.535383 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:39.556285 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:39.567054 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:39.571478 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:39.669265 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:15:39.829653 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:15:39.875604 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:15:39.903152 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:39.925926 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:39.945959 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:40.005103 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:40.031553 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:40.032845 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:40.048979 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:40.050331 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:40.057173 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:40.057921 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:15:40.072314 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:40.074294 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:40.114135 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:40.135620 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:40.142529 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:40.168232 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:40.186733 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:40.258183 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:15:40.271337 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:40.349591 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:40.362665 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:40.382287 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:40.391664 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:40.401732 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:40.428925 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:40.430889 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:40.432368 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:40.524745 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:40.602787 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:40.615583 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:40.621053 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:40.646163 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:40.674998 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:40.695741 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:40.712944 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:40.753062 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:15:40.783921 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:15:40.795696 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:15:40.844111 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:15:40.963310 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:15:40.973047 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:15:41.132342 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:41.235705 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:41.242292 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:41.325274 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:41.333469 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:41.377626 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:41.406540 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:41.469776 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:41.478538 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:41.488055 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:41.508456 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:41.510407 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:41.537312 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:41.563278 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:41.579553 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:41.625373 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:15:41.666759 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:15:41.691097 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:41.719244 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:41.730898 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:41.754972 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:41.780964 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:41.789856 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:41.791277 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:41.794831 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:41.795588 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:41.799515 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:41.801251 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:41.812144 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:41.857583 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:41.937458 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:42.028932 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:42.057514 27200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:46843, user_credentials={real_user=slave}} blocked reactor thread for 390742us
W20240501 20:15:42.099570 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:15:42.178495 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:42.251462 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:15:42.348311 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:42.350979 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:42.362429 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:42.363020 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:42.374991 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:42.393326 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:42.397698 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:42.464419 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:42.517632 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:15:42.641023 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:15:42.646010 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:42.680967 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:15:42.747678 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:15:42.792933 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:42.813066 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:15:42.850137 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:15:42.872481 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:42.896064 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:42.896705 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:42.921844 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:42.927222 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:42.997493 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:42.999914 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:43.029467 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:43.033643 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:43.037297 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:43.059629 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:43.070771 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:43.079164 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:43.117800 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:43.124394 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:43.179219 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:15:43.245424 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:15:43.298310 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:43.299103 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:43.322759 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:43.359135 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:43.406522 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:43.411906 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:15:43.417667 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:43.431331 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:43.433822 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:43.663522 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:43.692092 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:43.717439 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:43.743499 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:43.751623 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:43.775400 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:43.821379 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:43.825345 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:43.826071 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:43.856566 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:43.865022 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:43.865985 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:43.879346 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:43.884702 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:43.906772 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:43.932320 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:15:43.955502 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:15:44.290818 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:44.316433 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:44.355150 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:44.382131 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:44.392277 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:44.420282 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:15:44.430517 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:15:44.433328 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:15:44.463691 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:44.488638 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:44.515946 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:44.537722 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:44.547571 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:44.566064 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:44.597050 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:44.610211 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:44.645355 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:44.683265 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:44.716598 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:44.801455 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:15:44.805437 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:44.820297 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:44.840711 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:44.884863 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:44.888773 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:44.918164 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:44.921530 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:44.925256 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:44.989930 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:15:44.994305 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:15:45.048619 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:15:45.070204 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:45.147246 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:45.148029 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:45.234606 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:15:45.238189 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:45.251979 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:45.284451 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:45.284539 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:45.338572 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:45.418687 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:15:45.507726 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:15:45.521433 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:15:45.529294 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:45.552423 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:45.593869 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:45.685257 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:45.720911 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:45.750386 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:15:45.795886 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:45.819972 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:45.851269 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:15:45.901394 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:15:45.925132 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:15:45.952741 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:45.973023 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:45.978874 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:45.994217 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:45.997881 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:15:46.000419 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:46.021991 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:46.057889 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:46.067395 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 56820.2us
I20240501 20:15:46.101509 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:46.134393 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:46.188313 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:46.207191 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:46.211002 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:46.213395 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:46.239506 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:46.247198 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:46.359596 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:46.387209 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:46.388880 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:46.408773 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:46.420226 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:46.461508 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:46.481371 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:46.493925 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:46.497807 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:46.498193 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:46.642506 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:46.662564 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:46.673233 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:46.681785 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:46.710824 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:46.718139 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:46.738051 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:46.820238 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:15:47.084506 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:47.307896 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:47.317839 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:47.335234 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:47.337100 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:47.382552 27329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 80694.3us
W20240501 20:15:47.414139 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:47.416631 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:47.452642 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:47.454447 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:47.454671 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:47.467058 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:47.478886 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:47.481870 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:47.527724 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:47.527824 29049 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:47.530864 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:47.552426 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:47.562551 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:47.563747 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:47.603960 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:15:47.609009 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:47.630254 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:47.632578 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:47.635943 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:47.670893 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:47.709401 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:47.775712 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:47.781214 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:47.795864 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:47.799330 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:47.805030 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:47.828824 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:47.839982 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:47.850164 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:47.850925 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:47.930748 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:15:47.933657 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:48.000185 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:48.011211 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:15:48.043749 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:48.130061 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:15:48.324352 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:48.454841 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:48.478037 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:48.482900 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:48.487035 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:48.501999 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:48.502267 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:48.511142 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:48.603791 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:48.605558 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:15:48.607363 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:48.607358 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:48.613473 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:48.641445 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:48.697724 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:48.732021 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:48.741338 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:48.753803 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:48.866325 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:48.887212 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:48.887253 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:48.898895 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:48.919497 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:48.920281 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:48.928982 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:48.947340 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:49.051404 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:49.085994 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:49.087095 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:49.106153 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:49.114670 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:49.159425 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:49.178926 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:49.195670 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:49.435168 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:15:49.459285 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:15:49.460209 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:49.498396 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:49.539433 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:49.557953 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:49.600059 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:49.626453 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:49.668491 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:49.696501 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:49.698770 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:49.726866 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:49.754205 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:49.776932 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:49.804152 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:49.818892 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:49.825982 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:49.832196 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:49.832707 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:49.837930 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:49.893013 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:49.895849 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:49.909052 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:49.937319 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:49.946290 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:49.990392 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:15:49.996868 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:50.008275 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:50.010675 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:15:50.027376 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:15:50.088969 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:15:50.222522 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:15:50.241644 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:50.371487 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:50.403764 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:50.439692 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:50.468070 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:50.542857 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:50.543920 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:50.572358 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:50.572418 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:50.669797 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:50.706097 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:15:50.721911 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:50.725647 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:50.728761 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:50.746551 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:50.787722 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:50.809366 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:50.823632 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:50.825834 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:50.853901 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:50.859656 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:50.884981 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:50.885344 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:50.904624 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:15:50.946595 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:50.949175 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:50.956601 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:50.989110 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:51.123425 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:15:51.150449 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:15:51.151501 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:51.168329 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:51.171396 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:51.179457 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:51.197477 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:51.201087 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:51.203610 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:51.256870 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:51.386799 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:15:51.413326 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:15:51.642946 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:51.650607 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:51.655226 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:51.668071 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:51.690297 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:51.707828 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:51.724550 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:51.736403 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:51.797227 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:51.900490 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:51.932829 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:51.943271 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:51.946060 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:52.007138 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:52.010511 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:52.034150 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:52.051815 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:52.052959 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:52.077965 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:52.083148 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:52.128603 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:52.139182 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:52.192492 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 51160.5us
W20240501 20:15:52.219357 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:52.236326 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 49552.7us
I20240501 20:15:52.274159 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:52.290346 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:52.318583 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:52.391813 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:15:52.530290 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:52.547914 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:52.558434 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:52.600728 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:52.605116 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:52.613539 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:52.626610 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:52.642326 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:52.717649 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:52.770555 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:52.786065 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:52.806882 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:52.908679 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:52.923377 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:52.967434 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:53.000167 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:53.034771 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:53.056238 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:53.092564 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:53.244889 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:15:53.281471 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:15:53.292951 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:53.298873 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:53.348802 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:53.413797 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:53.413959 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:53.505105 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:53.526607 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:53.566820 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:53.603518 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:53.628733 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:53.681519 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:53.759116 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:53.775343 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:53.801465 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:53.814455 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:53.820122 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:53.835579 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:53.848260 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:53.864190 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:15:53.890172 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:53.898097 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:53.958868 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:53.973218 27330 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:46843, user_credentials={real_user=slave}} blocked reactor thread for 109041us
W20240501 20:15:54.049705 27202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 59305.9us
I20240501 20:15:54.063320 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:54.070586 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:54.103449 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:54.105121 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:54.180809 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:54.187166 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:54.191246 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:54.233644 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:54.320562 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:54.347028 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:54.404258 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:54.443104 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:54.445535 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:54.459185 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:54.462447 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:54.531989 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:15:54.533149 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:15:54.593600 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:15:54.627111 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:15:54.628803 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:54.662401 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:54.695047 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:54.734603 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:54.743731 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:54.753723 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:54.787532 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:54.788223 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:54.791291 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:54.978152 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:54.993356 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:54.998917 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:54.999608 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:55.010761 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:55.038049 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:55.073239 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:55.086239 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:55.187286 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:15:55.207453 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:15:55.222330 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:15:55.248605 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:55.256686 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:55.257299 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:55.270385 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:55.309444 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:55.313550 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:55.324553 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:55.362341 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:55.468750 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:55.569218 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:55.588982 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:55.601837 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:55.607502 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:55.621083 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:55.685948 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:55.718765 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:55.722884 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:55.773061 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:15:55.793552 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:55.905606 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:55.919167 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:55.931593 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:55.949983 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:55.971892 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:55.995432 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:56.000813 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:56.013222 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:15:56.136111 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:15:56.159013 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:56.177268 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:56.179486 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:56.202909 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:56.206912 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:56.214869 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:56.218806 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:56.223794 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:56.258291 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:15:56.290341 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:15:56.369323 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:15:56.527478 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:15:56.565733 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:56.588136 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:56.609683 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:56.633332 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 73961.5us
I20240501 20:15:56.636005 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:15:56.696466 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:56.698902 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:56.846294 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:56.882292 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:56.948714 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:15:56.955883 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:56.976529 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:15:56.989444 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:56.990598 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:57.005038 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:57.031410 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:57.034795 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:57.039582 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:57.150724 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:15:57.155611 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:57.224903 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:15:57.273108 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:57.283838 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:57.298907 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:57.299392 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:57.305277 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:57.336149 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:57.352265 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:57.461809 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:57.472965 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:57.490474 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:57.521648 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:57.551466 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:57.565440 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:57.566198 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:57.575541 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:57.648558 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:15:57.672235 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:15:57.718259 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:15:57.730849 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:15:57.850239 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:57.931983 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:57.934680 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:57.950376 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:57.966701 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:57.981174 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:57.982784 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:15:57.986670 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:15:58.055505 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:58.104631 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:58.135237 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:58.151396 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:58.171377 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:58.179572 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:58.182566 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:58.216305 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:15:58.221036 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:58.260879 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:58.398568 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:58.414870 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:58.427331 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:58.439349 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:58.482807 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:58.488873 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:58.495263 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:58.600484 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:15:58.685182 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:58.709486 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:58.714594 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:58.743472 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:58.745298 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:58.753808 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:15:58.761795 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:58.789395 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:58.821607 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:59.033512 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:15:59.131909 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:59.134279 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:59.162796 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:59.177575 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:59.180794 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:59.203146 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:59.230111 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:15:59.266682 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:59.274199 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:59.284896 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:59.292809 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:59.318291 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:59.343924 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:15:59.371192 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:59.399338 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:15:59.447465 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:15:59.512887 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:59.525488 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:59.546489 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:15:59.561966 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:15:59.591266 27068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 63899.3us
W20240501 20:15:59.599177 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:15:59.628628 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:15:59.635686 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:15:59.637616 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:15:59.637741 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:15:59.672664 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:15:59.673319 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:59.701963 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:59.715807 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:15:59.738976 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:15:59.904428 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:15:59.918229 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:15:59.931346 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:15:59.945076 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:15:59.950903 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:15:59.976217 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:15:59.995447 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:00.001762 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:00.072580 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:00.082912 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:00.095202 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:00.124905 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:00.138259 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:00.157397 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:00.159214 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:00.176537 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:00.178593 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:00.191273 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:16:00.255156 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:16:00.312605 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:16:00.356436 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:16:00.376591 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:00.405349 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:00.431265 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:00.436647 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:00.455641 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:16:00.485715 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:00.495497 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:00.542933 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:00.611918 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:00.635018 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:00.685840 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:00.695029 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:00.729312 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:00.763870 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:00.765164 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:00.770485 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:00.998448 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:01.106503 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:01.152854 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:01.153126 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:01.156102 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:01.158306 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:01.177855 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:01.228605 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:01.246500 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:01.262858 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:01.274353 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:01.275599 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:01.294394 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:01.327296 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:01.356729 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:01.367446 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:01.372342 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:01.411186 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:01.497465 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:01.523847 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:01.596668 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:01.609334 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:01.624148 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:01.650969 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:01.653728 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:01.685667 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:16:01.703115 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:01.712596 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:01.810108 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:01.838150 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:01.859822 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:16:01.859957 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:01.871614 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:01.873521 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:01.905550 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:01.910885 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:01.944720 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:02.000819 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:02.099344 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:02.126390 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:02.159579 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:02.197171 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:02.198781 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:02.228626 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:02.235630 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:02.273022 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:02.275543 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:02.389384 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:02.608574 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:02.612206 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:02.617014 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:02.639957 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:02.696345 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:02.713135 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:02.721833 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:02.733269 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:02.764570 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:16:02.781939 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:02.783161 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:16:02.791307 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:02.794711 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:02.800190 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:02.803808 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:02.805852 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:02.808544 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:02.861021 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:02.866674 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:16:02.876344 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:16:02.907469 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:16:03.086509 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:03.097445 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:03.097642 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:03.125586 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:16:03.126214 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:03.139365 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:03.180125 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:03.204980 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:03.285375 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:03.482288 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:03.583719 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:03.614161 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:03.625406 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:03.653959 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:03.654062 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:03.662737 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:03.699335 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:03.728008 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:03.893972 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:03.923046 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:03.931732 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:03.982767 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:16:04.023044 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:04.052829 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:04.054934 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:04.073712 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:04.130864 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:04.139622 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:04.183661 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:04.248085 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:04.248982 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:04.270751 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:04.277281 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:04.279816 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:04.305748 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:04.328059 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:04.357860 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:04.506991 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 68493.8us
I20240501 20:16:04.563426 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:04.573661 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:04.579402 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:04.581718 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:04.590253 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:16:04.595150 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:04.607084 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:04.663043 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:04.667552 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:04.668758 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 20:16:04.689625 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:04.690498 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:16:04.710131 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:04.856340 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:04.904404 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:04.915299 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:04.928408 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:04.953850 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:04.983609 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:05.004282 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:05.061519 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:05.109824 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:05.114070 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:05.132925 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:05.149019 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:05.151098 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:05.160751 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:05.171944 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:16:05.230332 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:05.279755 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:05.287529 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:05.368016 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:05.394366 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:05.407532 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:05.407588 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:16:05.426785 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:05.451769 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:05.463300 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 46604.7us
I20240501 20:16:05.508790 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:05.547185 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:05.582989 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:05.631201 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:05.656261 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:05.724550 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:05.773882 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:05.775038 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:05.798413 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:05.804025 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:05.816871 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:05.817391 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:05.820966 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:05.997534 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:06.006058 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:06.018865 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:06.035744 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:06.040113 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:06.052492 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:06.080076 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:06.168164 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:06.219715 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:06.227341 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:06.252317 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:06.264556 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:06.332060 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:16:06.345104 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:06.350081 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:06.354789 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:06.375381 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:06.423825 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:16:06.575227 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:06.588364 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:06.606953 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:06.608525 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:06.646526 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:06.650501 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:06.665571 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:06.711220 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:06.721228 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:06.726287 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:06.733989 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:06.737324 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:06.748342 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:16:06.748759 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:06.768079 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:06.792235 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:06.816009 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:06.847965 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:06.875749 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:06.887897 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:07.055943 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:07.072763 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:07.117007 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:07.118940 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:07.136459 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:07.172576 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:07.182690 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:07.188614 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:07.234222 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:07.249578 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:07.275836 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:07.290994 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:07.304440 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:07.309623 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:07.320784 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:07.361428 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:07.419816 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:07.581936 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:16:07.681757 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:07.750481 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:07.758585 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:07.799386 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:07.821251 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:07.822149 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:07.829372 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:07.829954 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:07.848238 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:07.853438 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:07.894192 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:07.900815 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:07.919031 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:07.935920 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:07.940907 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:07.953928 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:16:07.970263 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.195:35443, user_credentials={real_user=slave}} blocked reactor thread for 61835.5us
W20240501 20:16:07.971030 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:07.981703 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:07.985272 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:08.032775 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:08.077601 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:16:08.135459 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:08.173890 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:08.176340 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:08.187084 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:08.223146 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:08.223902 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:08.233597 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:08.382808 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:08.413554 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:08.420444 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:08.424337 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:08.438910 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:08.480151 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:08.508491 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:08.557077 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:08.816406 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:16:08.965456 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:08.987481 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:09.022059 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:09.027513 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:09.046692 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:09.069427 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:09.080209 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:09.091801 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:09.174866 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:09.248450 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:09.259872 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:09.284649 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:09.307686 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:09.314677 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:09.316625 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:09.323599 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:09.328403 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:09.349517 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:09.350476 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:09.352026 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:09.353945 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:09.363886 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:09.397419 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:09.409451 27505 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:09.414536 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:09.420806 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:09.447396 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:09.482964 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:09.499279 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:09.503887 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:16:09.511991 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:09.533051 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:09.538014 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:09.573207 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:09.606591 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:09.612823 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:09.716039 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:16:09.815045 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:16:09.940141 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:16:10.007465 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:16:10.040489 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:10.124300 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:10.139542 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:10.147709 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:10.168190 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:10.216660 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:10.228102 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:16:10.233381 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:10.244254 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:10.272382 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:10.290356 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:10.331218 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:10.407020 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:10.505659 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:10.508265 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:10.517946 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:10.523556 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:10.528954 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:10.538615 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:10.550638 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:10.570317 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:10.577549 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:10.606694 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:10.618894 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:10.622362 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:10.632875 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:10.683936 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:10.741852 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:10.743808 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:10.874912 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:10.901803 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:10.918888 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:10.925673 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:10.947701 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 81461.5us
I20240501 20:16:10.959270 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:10.995961 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:11.011464 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:11.015973 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:11.250269 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:11.281463 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:11.299096 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:11.299088 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:11.299237 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:11.325203 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:16:11.347815 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:11.359277 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:11.499090 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:16:11.574429 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:11.612030 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:11.623489 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:11.646787 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:11.665369 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:11.666741 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:11.684989 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:16:11.738096 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:11.747246 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:11.750473 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:11.790840 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:11.812561 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:16:11.815026 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:11.832824 29446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:11.833372 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:11.854398 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:11.855020 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:11.872921 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:11.891817 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:11.902254 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:11.922087 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:16:11.948055 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:16:11.968307 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:12.126111 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:12.127915 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:12.147576 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:12.155472 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:12.166059 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:12.186787 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:12.202761 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:12.205691 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:12.428287 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:12.464907 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:12.471534 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:12.503036 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:12.503201 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:12.504072 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:12.567288 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:12.569348 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:12.599642 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:12.679154 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:16:12.683759 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:16:12.701875 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:12.878150 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:12.931216 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:12.931375 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:12.938136 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:12.958942 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:12.971503 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:12.994875 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:13.013468 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:13.040709 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:13.061935 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:13.105674 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:13.150847 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:13.179118 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:13.212221 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:13.213642 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:13.233618 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:13.235766 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:13.260831 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:13.266515 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:13.299413 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:13.312796 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:13.333570 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:13.370086 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:13.381600 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:13.417145 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:13.478232 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:13.573459 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:13.592944 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:13.596486 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:13.603325 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:13.673501 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:13.676277 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:13.766515 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:13.875556 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:16:14.013394 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:14.024449 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:14.052510 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:14.055874 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:14.062381 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:14.075400 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:14.114277 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:14.130463 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:14.138260 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:14.268874 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:16:14.411202 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:16:14.412811 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:14.460851 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:14.475978 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:14.507450 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:14.534598 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:14.536468 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:14.550019 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:14.567137 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:14.581231 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:14.588380 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:14.611546 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:14.625258 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:14.638818 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:14.693270 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:14.697000 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:14.727208 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:14.732683 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:14.737619 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:16:14.743324 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:16:14.728523 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:16:14.764199 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:14.790643 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:14.793900 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:14.808681 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:14.823206 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:14.833132 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:14.834178 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:14.860209 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:16:14.890574 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:15.040434 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:16:15.076339 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:16:15.228909 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:15.239889 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:15.256783 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:15.308593 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:15.312656 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:15.316812 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:15.356709 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:15.356887 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:15.389403 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 123851us
W20240501 20:16:15.400234 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:15.483402 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:15.503362 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:15.529745 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:15.584175 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:15.592550 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:15.614231 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:15.616907 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:15.627702 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:15.638900 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:15.705004 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:15.720698 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:16:15.960222 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:15.972066 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:15.980177 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:16.005437 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:16.013778 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:16.035888 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:16.087540 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:16.100522 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:16.117481 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:16.124475 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:16.134927 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:16.152096 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:16.206377 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:16.215267 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:16.244632 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:16.339139 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:16.369421 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:16.379665 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:16.391357 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:16:16.392722 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:16.397131 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:16.411039 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:16.451788 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:16.530735 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:16:16.570048 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:16:16.596601 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:16:16.599431 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:16:16.639828 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:16.647835 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:16.656450 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:16.664000 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:16.690891 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:16.713557 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:16.747730 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:16.748898 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:16:16.808221 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:16.982151 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:17.007086 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:17.007511 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:16:17.041231 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:16:17.047315 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:17.063481 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:17.080243 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:17.099843 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:17.103776 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:17.134856 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:17.187631 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:17.285655 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:16:17.300453 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:16:17.369262 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:17.385557 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:17.413203 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:17.414592 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:17.440012 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 61852.6us
W20240501 20:16:17.441314 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 65479.3us
I20240501 20:16:17.480958 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:17.485162 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:17.486449 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:17.527563 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:17.617164 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:17.674122 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:17.678879 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:17.695422 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:16:17.704282 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:17.737740 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:17.740690 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:17.746452 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:17.768900 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:17.799122 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:17.875196 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:17.875671 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:17.875808 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:17.877887 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:17.921650 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:17.979617 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:17.991904 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:18.021140 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:18.042146 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:18.114696 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:16:18.297973 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:16:18.390323 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:18.472256 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:18.476377 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:18.481348 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:18.489951 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:18.494282 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:18.516898 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:18.541505 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:18.784210 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:18.789744 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:18.796052 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:18.808033 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:18.823427 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:18.842933 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:18.880740 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:18.885708 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:19.054240 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:16:19.063262 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:19.064039 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:19.069137 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:16:19.078588 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:19.107131 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:19.121807 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:16:19.132400 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:19.158480 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:19.178575 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:19.226030 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:16:19.234334 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 62016.7us
W20240501 20:16:19.237788 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:16:19.239073 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:16:19.338084 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:19.416862 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:19.432938 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:16:19.467100 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:19.468947 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:19.492036 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:19.567662 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:19.577548 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:19.583806 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:19.675226 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:16:19.756450 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:16:19.760354 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:19.804420 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:19.805377 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:19.814282 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:19.849177 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:16:19.851368 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:19.902765 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:19.917235 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:19.964900 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:16:19.969753 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:19.973017 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:19.973882 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:20.025739 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:20.035722 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:16:20.070888 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:20.077116 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:20.087411 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:20.112640 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:20.115831 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 20:16:20.154867 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:16:20.190423 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:16:20.211588 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:20.425786 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:20.449678 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:20.455278 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:20.473462 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:20.481211 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:20.482995 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:20.490926 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:20.492249 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:20.531504 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:20.557490 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:20.607111 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:20.633905 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:20.674135 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:20.674189 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:16:20.712445 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:20.716006 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:20.723292 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:20.980151 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:20.986467 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:21.013242 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:21.044128 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:21.062215 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:21.075784 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:21.094383 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:21.265304 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:21.277503 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:21.285744 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:21.313534 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:16:21.313640 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:21.330860 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:21.342558 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:21.385129 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:21.488849 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:21.496624 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:21.505972 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:21.526424 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:21.529026 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:21.552547 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:21.555162 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:21.577365 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:21.603663 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:21.606032 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:21.606053 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:21.666694 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:21.686923 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:16:21.742125 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:16:21.747710 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:21.754653 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:21.864714 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:21.914597 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:21.918126 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:21.925419 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:21.942180 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:21.966853 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:21.997251 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:22.133169 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:22.196245 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:22.229005 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:22.229966 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:22.256022 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:22.289754 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:22.297468 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:22.329247 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:22.332320 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:22.355123 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:22.417968 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:16:22.421774 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:16:22.616096 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:22.658394 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:22.659463 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:16:22.685497 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:22.689334 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:16:22.690155 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:22.701146 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:22.711742 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:22.753504 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:22.766386 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:22.776198 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:22.779222 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:22.793426 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:22.795189 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:22.869859 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:22.873562 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:22.894557 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:22.920671 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:16:23.069099 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:16:23.120842 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:23.130892 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:23.150712 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:23.171473 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:23.173243 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:23.181751 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:23.205053 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:23.233273 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:23.248813 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:23.399255 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:23.449081 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:23.472707 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:23.496641 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:23.529328 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:23.548595 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:23.548804 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:23.619904 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:23.751155 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:16:23.855473 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:23.908217 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:23.926214 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:23.932345 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:23.936743 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:23.942870 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:23.982162 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:23.986231 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:16:23.995970 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:16:24.021124 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:24.033586 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:24.066130 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:24.068068 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:24.069178 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:24.072377 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:24.119277 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:16:24.144483 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:16:24.153043 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:16:24.223299 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:24.236197 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:16:24.247747 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:24.513655 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:24.519685 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:24.542462 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:24.545848 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:24.553484 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:24.566428 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:24.578647 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:24.608248 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:24.659327 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:16:24.678536 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:24.732102 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:24.765802 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:24.766285 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:24.788762 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:16:24.789940 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:24.850015 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:24.854210 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:24.873850 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:16:24.895452 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:16:24.906890 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:24.970206 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:25.004894 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:25.006304 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:25.034404 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:25.047806 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:25.064778 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:25.096510 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:25.273675 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:16:25.277525 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:16:25.278707 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:16:25.319090 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:16:25.367043 27460 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:46843, user_credentials={real_user=slave}} blocked reactor thread for 93370.2us
I20240501 20:16:25.388526 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:25.397361 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:25.404217 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:25.411859 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:25.413203 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:25.436781 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:25.530892 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:25.597626 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:25.605331 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:16:25.766206 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:25.784253 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:25.794077 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:25.816391 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:25.849982 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:25.853931 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:16:25.917325 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:25.994714 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:26.132987 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:26.145145 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:26.163157 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:26.166121 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:26.179930 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:26.197726 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:26.201160 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:26.222508 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:26.232503 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:26.244298 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:26.277383 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:26.288017 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:26.303794 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:26.323719 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:26.328832 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:26.345021 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:26.355852 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:26.489704 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:16:26.509888 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:16:26.568548 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:26.596591 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:26.619158 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:26.625758 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:26.648787 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:26.654147 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:26.674540 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:26.693945 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:26.765162 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:16:26.766074 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:16:26.825040 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:16:26.888080 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:26.899739 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:26.913405 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:26.950206 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:26.966231 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:26.988384 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:27.038465 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:27.046595 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:27.155478 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 67475.6us
W20240501 20:16:27.175796 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:16:27.178473 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:27.237671 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:27.348886 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:27.379285 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:27.379515 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:27.392840 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:27.402899 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:27.404183 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:16:27.462395 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:16:27.476565 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:27.504205 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:27.564437 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:27.564936 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:27.598335 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:27.639833 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:27.640923 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 20:16:27.721545 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:27.749389 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:27.761777 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:27.765996 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:27.821313 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:27.852545 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:27.865574 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:27.867916 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:27.887037 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:27.910670 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:27.946681 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:27.963845 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:28.016408 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:16:28.020195 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:16:28.143610 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:28.151247 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:28.168437 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:28.175244 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:28.178467 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:28.190840 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:28.215458 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:28.219157 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:28.220134 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:28.369202 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:28.391775 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:28.414259 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:28.527690 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:28.532538 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:28.560210 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:28.637780 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:28.652015 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:28.740309 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:16:28.801604 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:16:28.832857 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:16:28.886829 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:16:28.898227 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:28.912789 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:28.930685 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:28.951380 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:28.963757 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:29.000453 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:29.021709 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:29.022236 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:29.165958 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:29.183493 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:29.195211 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:29.218513 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:29.228264 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:29.257696 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:29.263370 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:29.266323 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:29.270182 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:29.449790 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:29.474078 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:29.530545 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:29.593322 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:29.610177 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:29.638886 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:29.668188 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:29.700688 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:29.718122 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:29.786104 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:29.792977 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:29.797447 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:29.804302 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:29.819857 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:29.822293 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:29.830714 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:29.837955 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:29.857748 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:29.946599 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 20:16:29.969795 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:16:30.063463 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:30.091339 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:30.109544 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:30.121130 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:30.129069 27114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:30.129616 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:30.134577 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:30.201043 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:30.209285 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:30.259788 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:30.300232 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:30.306080 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:30.309345 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:30.340066 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:16:30.367215 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:30.371529 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:30.372270 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:30.497895 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:16:30.564793 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:16:30.604717 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:30.676259 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:30.685981 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:30.699059 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:30.699682 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:30.704643 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:30.709662 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:30.724932 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:30.733958 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:30.799436 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:30.803084 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:30.813382 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:30.829170 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:30.847723 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:30.874222 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:30.886337 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:30.903324 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:31.221683 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:16:31.230305 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:31.239642 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:31.286270 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:31.288952 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:31.299232 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:31.304457 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:31.342832 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:31.356406 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:31.357445 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:31.372821 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:31.428911 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:31.457768 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:31.513367 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:16:31.542383 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:16:31.569247 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:31.588451 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:31.605911 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:31.613746 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:31.643905 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:31.687322 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:16:31.725273 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:31.756075 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:31.772928 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:31.807884 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:31.816141 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:31.826745 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:31.866015 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:31.889861 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:31.907665 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:31.912087 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:32.203564 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:32.233796 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:32.270809 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:32.271553 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:32.295559 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:32.308303 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:32.335193 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:32.341135 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:32.371899 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:32.449465 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:32.451457 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:32.475451 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:32.486617 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:32.500301 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:32.605574 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:32.624774 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:32.629935 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:32.634362 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:32.663740 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:32.729231 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:16:32.730285 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:32.751247 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:32.755836 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:32.780341 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:32.781558 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:32.782071 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:32.783330 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:32.816164 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:32.859910 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:32.950493 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:32.959460 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:32.967998 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:32.976315 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:32.978782 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:32.987286 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:33.014453 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:33.026870 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:33.087363 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:33.328811 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:33.346302 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:33.350690 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:33.414377 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:33.438845 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:33.459475 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:33.469514 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:33.490578 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:33.512326 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:33.594372 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:33.639180 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:33.696835 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:33.734735 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:33.751798 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:33.763931 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:33.803468 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:33.824836 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:33.833603 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:33.898142 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:16:33.908102 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:16:33.925900 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:33.947162 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:33.958546 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:33.959642 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:33.974987 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:33.979316 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:34.017371 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:34.048637 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:34.050333 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:34.337893 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:34.338235 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:34.351840 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:16:34.354997 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:34.363687 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:34.374609 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:34.389139 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:34.406566 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:34.445298 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:34.498250 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:34.663596 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:34.727097 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:34.731642 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:34.732053 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:34.776607 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:34.778990 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:34.805297 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:34.809885 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:34.820930 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:34.922528 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:34.927460 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:34.954921 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:34.957499 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:34.978906 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:35.000316 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:35.002554 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:35.010236 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:16:35.026007 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:16:35.032763 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:35.230571 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:35.243871 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:16:35.244828 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:35.248155 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:35.255051 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:35.272303 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:35.288718 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:35.303638 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:35.354481 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:16:35.363946 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:35.374979 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:35.379899 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:35.389566 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:35.417789 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:35.442920 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:35.444592 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:35.460857 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:35.490687 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:35.490860 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:35.666142 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:16:35.898685 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:16:35.954766 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:36.021253 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:36.040038 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:36.138064 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:36.148730 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:36.165084 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:36.168171 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:36.185885 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:36.200268 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:36.221571 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:36.248425 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:36.283854 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:36.290253 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:36.298864 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:36.307626 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:36.355155 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:36.387847 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:16:36.400632 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:16:36.448184 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:16:36.449363 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:36.488626 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:36.601081 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:36.608500 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:36.629001 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:36.631198 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:36.660859 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:16:36.693012 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:36.771561 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:36.773816 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:36.843621 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:36.868197 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:36.892692 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:36.932152 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:36.935653 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:36.943285 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:36.985384 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:37.023140 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:37.041533 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:37.066771 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:37.265224 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:37.303472 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:37.350621 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:37.366683 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:37.384594 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:37.388135 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:37.422238 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:37.440040 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:37.456276 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:37.461941 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:37.471657 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:37.513016 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:37.521420 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:37.530957 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:37.580425 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:37.584467 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:37.590216 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:37.612897 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:37.715713 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:37.749385 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:16:37.802898 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:37.825585 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:37.836370 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:37.847963 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:37.856815 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:16:37.866036 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:37.908601 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:37.913769 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:37.917708 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:37.956802 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:38.015622 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:38.023778 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:38.037495 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:38.068737 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:38.073899 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:38.109580 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:38.115532 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:38.136778 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:38.333253 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:16:38.367089 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:16:38.446101 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:38.464061 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:38.478824 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:38.522002 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:38.530901 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:38.544488 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:38.596685 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:38.610821 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:38.791273 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:38.791693 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:38.792243 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:38.794168 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:38.797132 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:38.840024 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:38.843744 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:38.857378 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:38.884246 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:38.889369 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:38.900569 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:38.939316 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:38.980284 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:39.007855 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:39.016924 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:39.026829 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:16:39.078534 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:39.147634 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:39.181746 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:39.282658 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:39.403156 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:16:39.421144 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:39.428944 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:39.451793 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:39.452929 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:39.515684 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:39.520709 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:39.555369 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:39.557197 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:39.622248 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:39.654987 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:39.708956 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:39.726286 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:39.769766 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:39.806202 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:39.809273 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:39.833225 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:39.857884 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:39.870774 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:39.870826 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:39.908852 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:39.935042 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:39.936899 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:39.945808 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:40.002905 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:16:40.027791 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:16:40.083313 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:40.122566 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:40.181674 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:40.217453 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:40.256225 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:16:40.262348 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:40.270483 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:40.311378 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:40.324777 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:16:40.335018 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:40.359280 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 20:16:40.369463 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:40.412400 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:40.503993 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:40.568336 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:40.602397 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:16:40.602856 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:40.656672 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:40.668430 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:40.683794 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:40.708840 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:40.745805 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:40.749657 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:40.772902 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:40.933246 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:40.979033 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:40.979115 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:40.979780 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:40.995234 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:41.002522 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:41.011788 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:41.030987 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:41.034808 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:41.056913 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:41.064924 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:41.094324 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:41.098881 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:41.154678 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:41.158191 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:41.400081 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:41.484951 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:41.520911 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:41.527714 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:41.547456 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:16:41.551489 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:41.553332 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:41.556396 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:41.587431 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:41.587989 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:41.609058 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:41.629263 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:41.631788 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:41.682888 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:41.694062 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:41.726449 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:41.792060 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:41.815217 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:41.831068 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:41.851696 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:16:41.960003 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:41.982117 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:41.994714 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:42.022372 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:42.053050 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:42.082424 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:42.121332 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:16:42.135047 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:42.154198 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:42.180423 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:42.218194 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:42.243319 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:42.261509 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:42.263278 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:42.275866 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:42.303030 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:42.329999 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:42.336488 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:42.443547 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:42.698547 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:42.710862 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:16:42.724205 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:42.852890 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:42.870435 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:42.896225 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:42.904917 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:42.940768 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:16:42.944943 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:42.965373 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:42.991912 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:42.994644 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:43.013898 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:43.019266 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:43.021082 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:43.033500 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:43.051393 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:43.064004 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:43.090523 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:43.091745 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:43.116221 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:16:43.158447 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:43.230801 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:43.244230 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:43.284279 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:16:43.287672 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:43.298429 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:43.307739 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:16:43.312896 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:43.328100 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:43.336071 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:43.553910 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:43.554914 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:43.558141 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:43.598874 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:43.610196 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:43.641729 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:43.658562 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:43.677994 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:43.731377 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:43.736421 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:43.767346 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:43.773171 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:43.801543 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:16:43.812829 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:43.813158 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:43.818854 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:43.851836 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:44.005543 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:44.143959 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:44.163967 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:44.179620 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:44.184830 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:44.215391 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:44.218408 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:16:44.223405 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:44.248521 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:44.266506 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:44.299108 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:44.309901 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:44.310510 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:44.321120 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:44.352285 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:44.371719 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:44.396673 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:44.444528 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:44.486436 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 73576.2us
W20240501 20:16:44.603181 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:44.708287 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:44.760982 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:44.803752 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:44.857358 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:44.857410 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:44.867059 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:44.878610 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:16:44.896689 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:44.900959 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:44.925536 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:44.949106 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:44.975124 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:44.999877 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:45.013373 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:45.015906 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:45.040161 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:16:45.042251 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:16:45.043603 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:45.058424 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:45.113061 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:45.180328 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:45.206766 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:45.211377 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:45.231554 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:45.241446 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:45.264245 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:45.270609 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:45.297500 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:45.338645 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:45.373912 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:45.491765 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:16:45.583565 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:45.650441 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:45.661943 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:16:45.663844 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:45.688652 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:45.691990 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:45.731498 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:45.742739 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:45.745546 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:45.989208 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:16:46.007308 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:46.020184 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:46.028426 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:46.037673 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:46.064445 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:46.091461 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:46.102108 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:46.130774 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:46.147001 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:46.150698 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:46.153096 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:46.178263 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:46.183485 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:46.215145 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:46.215994 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:46.226387 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:46.242885 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:46.294128 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:46.310662 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:46.319223 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:46.344594 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:46.384429 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:46.394450 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:46.400839 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:46.434463 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:46.493170 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:46.551838 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:46.753859 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:46.763051 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:46.763958 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:46.825599 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:46.830359 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:46.834729 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:46.834976 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:46.844561 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:46.925539 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:16:46.931373 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:47.159096 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:47.210222 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:47.214395 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:47.256587 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:47.297955 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:16:47.304152 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:47.328466 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:47.336625 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:47.336896 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:47.361382 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:16:47.374006 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:16:47.392036 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:47.424443 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:47.429431 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:47.434758 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:47.438431 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:47.440546 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:47.445430 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:47.457589 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:47.457877 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:47.470649 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:47.502840 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:47.509325 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:47.543067 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:47.561748 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:47.627825 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:16:47.661087 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:16:47.850296 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:16:47.892552 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:16:47.904413 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:47.911375 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:47.937077 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:47.974754 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:47.984107 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:47.991104 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:47.992192 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:48.057615 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:48.115792 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:48.117133 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:48.135160 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:48.163923 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:48.240106 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:48.248252 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:48.251511 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:48.268816 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:48.314004 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:48.387465 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:48.400732 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:48.428475 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:16:48.482266 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:48.493856 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:48.508625 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:48.538231 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:48.570674 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:48.607632 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:48.611310 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:48.616520 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:48.669919 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:48.755836 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:16:48.812858 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:48.814781 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:48.828313 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:48.880160 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:48.922134 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:48.953826 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:48.993407 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:48.994952 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:49.061146 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:49.231660 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:49.272553 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:16:49.305029 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:49.456893 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:49.457633 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:49.495064 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:49.495707 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 44644.7us
I20240501 20:16:49.533180 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:49.554088 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:49.567267 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:49.583357 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:49.616375 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:49.639220 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:49.678162 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:49.697685 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:49.704522 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:49.716773 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:49.755751 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:49.787225 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:49.826838 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:16:49.833375 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:49.891498 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:49.988276 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:50.006052 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:50.012651 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:50.013325 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:50.037204 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:50.040683 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:16:50.044512 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 20:16:50.062721 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:50.100890 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:50.262961 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:50.268195 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:50.271243 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:50.373059 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:50.377871 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:50.380013 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:50.385643 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:50.410821 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:50.450016 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:50.451637 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:50.495210 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:50.516068 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:16:50.576766 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:16:50.599081 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:50.789121 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:50.796574 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:50.813925 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:50.854648 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:50.946813 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:50.968732 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:50.990399 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:51.001168 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:51.010506 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:51.062556 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:51.067837 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:51.082562 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:51.103929 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:51.124647 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:51.131042 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:51.131420 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:51.139811 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:51.215916 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:51.223121 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:51.248708 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:51.249339 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:51.292086 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:51.314999 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:51.324311 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:51.341833 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:51.348629 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:16:51.380844 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:51.532948 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:51.576161 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:51.577394 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:51.599539 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:51.602933 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:51.610594 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:51.620435 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:51.697944 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:51.871696 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:51.872424 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:51.882895 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:51.896193 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:51.919020 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:51.928229 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:51.933604 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:51.956456 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:51.977087 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:52.091701 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:16:52.191110 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:52.209970 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:52.244800 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:52.257786 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:52.279456 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:52.282840 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:52.375856 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:52.383628 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:52.400977 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:52.405787 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:16:52.407480 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:52.420876 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:52.459955 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:52.478193 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:52.493197 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:52.511952 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:52.580799 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:52.603155 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:52.697499 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:16:52.742507 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:16:52.771494 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:16:52.824433 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:52.971742 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:52.992480 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:53.024204 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:53.048401 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:53.060752 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:53.128070 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:53.137545 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:53.140954 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:53.163885 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:53.188630 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:53.193825 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:53.247709 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:53.268013 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:53.329967 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:53.338165 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:53.371344 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:53.376250 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:53.445852 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:16:53.492502 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:53.496028 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:53.515676 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:53.544947 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:53.548476 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:53.551944 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 65363.5us
W20240501 20:16:53.576788 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:53.585367 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:53.591367 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:53.603736 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:53.710752 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:53.722086 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:53.751991 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:53.785192 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:53.815436 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:53.846433 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:16:53.878312 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:53.878865 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:53.879297 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:54.023627 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:54.046427 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:54.057534 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:54.070940 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:54.090320 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:54.107318 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:54.138248 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:54.154211 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:54.222970 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:54.230967 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:54.389313 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:54.398393 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:54.413329 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:54.430392 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:54.431991 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:54.449398 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:54.456902 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:54.569583 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:16:54.600919 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:16:54.644779 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:54.704229 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:16:54.719421 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:54.760708 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:54.788486 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:54.834338 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:54.881156 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:54.896203 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:54.900077 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:16:54.911881 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:54.918741 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:54.955368 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:54.972126 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:54.987480 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:55.009755 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:55.039302 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:16:55.063463 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 20:16:55.068712 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:55.083431 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:55.130352 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:55.139823 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:55.162400 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:55.233891 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:55.271067 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:55.329288 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:55.363317 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:55.365099 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:55.377555 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:55.407752 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:55.425983 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:55.485426 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:16:55.554003 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:55.587883 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:55.596277 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:55.609197 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:55.632753 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:55.652062 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 20:16:55.693964 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:55.701875 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:55.721948 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:55.748309 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:55.958473 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:55.988202 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:55.994098 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:56.010486 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:56.029676 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:56.038537 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:56.040298 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:56.070210 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:56.164945 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:16:56.322782 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:16:56.368907 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:56.382910 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:56.383515 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:56.385779 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:56.394203 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:56.398979 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:56.423363 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:56.466463 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:16:56.467352 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:56.501775 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:56.522904 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:56.526080 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:56.549652 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:56.563326 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:56.586881 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:56.617748 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:56.654990 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:16:56.661244 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:16:56.788059 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:56.846719 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:56.888852 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:56.890760 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:16:56.900790 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:56.930509 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:56.953544 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:56.956880 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:57.215490 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:57.223335 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:57.224920 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:16:57.233294 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:16:57.258088 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:57.275104 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:57.289958 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:16:57.298288 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:57.300123 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:57.344871 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:57.374213 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:16:57.419899 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:57.477783 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:16:57.530597 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:16:57.550295 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:16:57.567852 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:57.598415 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:57.604676 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:57.634827 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:57.647895 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:57.664376 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:57.798259 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:57.811765 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:57.812873 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:57.812913 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:57.815497 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:57.837340 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:57.842885 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 100806us
W20240501 20:16:57.866257 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:57.908807 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:57.908917 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:57.923548 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:16:57.958106 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:16:57.959661 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:16:58.090034 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:58.106438 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:58.132516 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:58.134605 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:58.155783 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:58.191063 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:58.192883 27068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 47096.3us
I20240501 20:16:58.199788 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:58.261296 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:58.478356 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:58.516976 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:58.518432 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:58.537129 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:58.548735 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:58.564144 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:58.576328 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:58.576750 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:58.622123 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:58.662992 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:58.680848 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:58.685809 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:58.755146 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:16:58.780390 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:58.797263 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:58.800887 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:58.841058 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:16:58.861116 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:16:58.913928 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:16:58.988183 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:16:59.025192 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:59.048894 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:16:59.060457 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:59.065383 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:59.112071 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:16:59.119004 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:59.167887 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:59.280011 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:16:59.335023 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:16:59.352792 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:59.377593 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:59.382993 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:59.383317 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:16:59.415798 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:59.423393 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:59.436321 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:59.457670 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:16:59.469497 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:16:59.642746 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:16:59.793148 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:59.820122 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:16:59.828493 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:16:59.829474 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:16:59.858475 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:16:59.859638 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:16:59.861418 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:16:59.879261 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:16:59.891471 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:16:59.907209 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:16:59.940611 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:16:59.947281 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:16:59.956022 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:16:59.985626 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:00.014150 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:00.053478 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:17:00.066529 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:00.092708 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:00.094532 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:00.109937 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:17:00.143738 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:00.256412 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:00.293774 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:17:00.423991 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:00.436981 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:00.465816 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:00.471987 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:00.486404 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:00.556221 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:00.591063 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:00.605474 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:00.669446 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:00.731240 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:00.744391 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:00.778362 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:00.793172 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:00.824729 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:00.841373 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:17:00.847065 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:00.919718 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:00.956426 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:00.971715 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:00.975931 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:00.994984 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:00.998868 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:01.034797 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:01.044023 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:01.249234 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:01.256031 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:01.273293 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:01.314469 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:01.342770 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:01.342985 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:01.371235 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:01.385152 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:01.426954 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:01.480072 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:17:01.480216 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:17:01.492091 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:01.730330 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:01.760561 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:01.786484 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:01.790506 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:01.826364 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:01.841181 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:01.886829 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:01.899418 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:01.903168 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:02.080127 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:17:02.099156 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:02.189086 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:02.209893 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:17:02.229696 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:02.263481 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:02.274900 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:02.292479 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:02.300743 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:02.308478 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:02.311691 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:02.313235 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:02.343180 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:02.347880 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:02.423223 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:02.437191 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:02.443058 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:02.462391 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:02.464565 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:02.476770 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:02.478730 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:02.484824 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:17:02.521906 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:02.533944 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:02.548302 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:02.554359 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:02.561383 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:02.611022 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:02.613819 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:02.658525 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:02.676195 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 47243.6us
W20240501 20:17:02.857690 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:02.970880 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:02.994819 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:03.002691 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:03.007997 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:17:03.018045 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:03.061576 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:03.088907 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:03.108923 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:03.113248 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:03.121508 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:17:03.142921 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:03.207851 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:03.211443 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:03.218165 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:03.223660 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:03.263447 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:03.263577 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:03.314194 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:03.489777 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:03.495810 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:03.496189 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:03.546891 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:03.582329 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:03.589525 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:03.595286 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:03.700788 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:03.806684 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:03.817888 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:03.850661 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:03.851414 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:03.886368 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:17:03.892791 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:17:03.924067 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:03.935076 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:04.001938 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:04.004009 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:04.018731 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:04.073673 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:17:04.150341 29315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:04.150965 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:04.191190 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:04.230239 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:04.243810 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:04.245909 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:04.246471 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:04.275837 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:04.299176 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:04.322911 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:04.343592 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:04.348968 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:04.368568 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:04.386286 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:04.387589 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:04.439039 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:04.443130 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:04.723096 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:17:04.763634 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:04.779513 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:04.789888 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:17:04.794685 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:04.824524 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:04.860931 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:04.920367 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:04.924728 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:04.941416 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:04.972434 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:04.981385 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:04.988786 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:17:05.021908 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:17:05.022215 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:05.055056 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:05.088771 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:05.115500 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:05.150785 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:05.192425 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:05.198508 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:17:05.200222 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:05.235291 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:05.274672 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:05.372045 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:17:05.497943 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:05.523255 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:05.562753 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:05.563719 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:05.584923 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:05.589799 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:05.610544 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:05.612365 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:05.628667 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:05.642158 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:17:05.651873 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:05.662407 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:05.670881 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:05.700014 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:05.702118 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:05.767143 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:05.959746 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:17:06.010880 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:06.033037 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:06.052089 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:06.059089 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:06.082408 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:06.088678 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:06.108521 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:06.245890 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:06.254738 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:06.302982 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:06.323092 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:06.359997 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:06.375370 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:06.377435 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:06.377799 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:06.380223 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:06.514312 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:17:06.522254 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:17:06.557335 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:17:06.737264 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:06.837733 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:06.856199 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:06.871554 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:06.923589 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:06.961167 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:06.969313 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:17:06.990375 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:07.001652 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:07.025207 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:07.027451 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:07.075145 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:07.079978 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:07.089972 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:07.096385 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:07.102692 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:17:07.140277 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:07.154479 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:07.158104 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:17:07.187429 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:07.187616 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:07.231339 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:07.255452 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:17:07.256981 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:17:07.257202 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:07.260813 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:07.346120 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:07.348222 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:17:07.387445 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:17:07.388990 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:07.461279 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:07.480610 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:17:07.482945 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:07.493508 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:07.540338 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:07.548636 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:07.579109 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:07.615912 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:07.720620 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:17:07.767181 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:07.899377 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:07.917495 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:07.938498 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:07.976553 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:07.980746 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:08.010936 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:08.018348 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:08.023373 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:08.128563 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:17:08.266706 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:08.316296 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:08.342129 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:08.357143 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:08.412755 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:08.427196 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:08.470445 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:08.479735 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:08.480823 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:08.613358 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:08.613621 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:08.630985 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:08.637873 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:08.658900 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:08.671407 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:08.711807 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:08.730170 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:08.753018 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:08.766330 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:08.766903 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:08.785454 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:08.807474 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:08.815346 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:08.912315 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:17:08.946892 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:08.975047 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:17:08.984405 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:17:09.029836 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:09.054126 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:09.062970 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:09.067685 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:09.088984 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:09.119235 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:09.130676 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:09.148841 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:09.192821 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:09.448671 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:09.503998 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:09.510475 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:09.524413 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:09.533452 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:09.623031 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:09.631778 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:09.657331 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:09.659332 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:09.661125 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:09.665982 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:17:09.698668 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 69116.1us
W20240501 20:17:09.701010 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:17:09.722906 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:09.745461 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:09.793284 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:09.794170 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:09.809834 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:09.844388 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:09.872287 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:09.882974 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:09.910490 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:09.912434 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:09.969945 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:09.987531 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:09.995429 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:10.024709 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:10.047739 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:10.048547 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:10.079948 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:10.088389 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:10.132953 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:10.239957 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:10.240180 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:10.265235 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:10.281332 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:10.326910 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:10.341759 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:10.356590 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:10.382719 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:10.390897 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:17:10.446419 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:17:10.469662 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:10.620215 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:10.646615 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:10.647965 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:10.687098 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:10.704030 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:10.710774 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:10.727599 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:10.746207 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:10.754137 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:10.755496 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:10.787770 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:10.788697 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:10.816545 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:10.867322 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:10.918493 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:11.004997 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:17:11.263161 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 20:17:11.289840 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:11.299031 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:11.309974 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:11.324730 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:11.327463 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:11.402683 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:11.416832 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:11.424568 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:11.429271 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:11.429903 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:11.446707 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:11.449103 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:11.465495 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:11.478654 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:11.480506 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:11.492862 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:11.498358 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:11.575598 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:11.576718 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:17:11.683254 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:11.692653 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:11.697363 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:11.732501 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:11.754109 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:11.769960 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:11.838191 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:11.873893 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:12.070458 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:12.084374 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:12.115859 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:12.126520 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:12.132553 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:12.149749 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:12.156944 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:12.161477 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:12.185817 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:12.193197 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:12.207216 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:12.210781 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:12.224553 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:12.229094 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:12.273391 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:17:12.301704 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:12.309654 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:12.328442 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:12.334645 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:12.369088 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:12.376343 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:12.389464 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:17:12.527743 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:17:12.588294 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:12.650151 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:12.654762 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:12.665177 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:12.683995 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:12.687642 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:12.715250 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:12.723577 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:12.734151 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:12.868894 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:12.880764 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:12.885221 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:12.897795 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:12.963263 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:12.968158 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:12.970054 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:13.096868 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:13.186223 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:13.202461 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:13.205377 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:17:13.216197 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:13.218959 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:13.227694 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:13.258742 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:13.320209 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:13.321233 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:13.543397 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:13.588817 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:13.598099 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:13.600562 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:13.612785 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:13.659075 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:13.674921 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:13.679210 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:13.795091 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:13.888329 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:13.902968 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:13.929279 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:13.942701 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:13.994833 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:13.998544 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:17:14.011313 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:14.038223 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:14.067028 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:14.068899 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:14.083566 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:14.095631 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:14.131098 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:14.131842 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:14.137059 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:14.153213 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:14.195434 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:14.233925 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:14.242820 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:14.252666 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:14.384763 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:17:14.427173 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:14.519227 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:14.540170 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:14.543792 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:14.560889 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:14.582391 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:14.607302 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:14.626657 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:14.639640 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:14.651005 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:17:14.681051 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:14.696513 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:14.740723 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:17:14.759183 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:14.769058 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:14.797140 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:14.811234 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:14.822902 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:14.873209 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:14.880561 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:14.893358 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:17:14.971812 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:15.078809 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:15.087553 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:15.123327 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:17:15.127578 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:15.130610 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:15.205241 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:15.206131 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:15.216249 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:15.235107 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:15.252123 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:15.297854 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:15.305606 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:15.335808 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:15.340550 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:15.342190 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:15.343381 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:15.502369 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:17:15.562413 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:15.563863 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:15.575603 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:15.594650 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:15.614720 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:15.635143 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:15.650702 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:15.729298 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:17:15.874533 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:16.006783 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:16.010558 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:16.018500 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:16.023324 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:16.050213 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:16.050243 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:16.064070 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:16.110211 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:16.110857 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:16.141271 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:16.171788 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:16.187793 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:16.276983 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:16.287086 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:16.290557 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:16.314683 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:17:16.336607 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:16.346000 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 71466.2us
W20240501 20:17:16.585093 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:16.598196 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:16.610396 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:16.614657 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:16.622300 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:16.683264 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:16.711531 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:16.748690 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:16.763713 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:16.771922 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:17:16.772562 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:17:16.789848 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:16.848067 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:16.849157 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:16.866611 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:16.892328 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:16.895668 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:16.906695 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:16.926206 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:16.929514 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:17.009018 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:17:17.187078 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:17.196399 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:17.215795 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:17:17.217406 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:17.225008 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:17.248623 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:17.265273 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:17.294529 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:17.303088 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:17.312445 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:17.314450 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:17.339897 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:17.351197 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:17.361636 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:17.411682 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:17.435171 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:17.439093 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:17.477985 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:17.494807 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:17.514832 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:17.532697 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:17:17.604135 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:17:17.754256 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:17.881664 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:17.881649 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:18.002442 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:18.009137 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:18.020445 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:18.023860 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:18.030297 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:18.030709 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:18.050467 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:18.066604 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:18.151377 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:18.155508 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:18.175316 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:18.221755 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:18.231374 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:18.242870 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:18.248189 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:18.318807 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:18.327802 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:18.329298 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:18.333973 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:18.372900 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:18.432026 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:17:18.440613 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:17:18.567274 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:18.593674 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:18.596643 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:18.597589 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:18.615152 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:18.622335 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:18.640686 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:18.662382 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:18.738565 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:18.956779 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:18.980079 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:18.996443 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:19.000805 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:19.001183 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:19.045395 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:19.049376 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:19.060353 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:19.095474 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:17:19.145059 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 20:17:19.275182 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:19.275761 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:19.290859 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:19.302986 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:19.330278 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:19.355669 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:19.406703 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:19.449290 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:19.465720 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:19.501610 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:19.516690 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:19.517366 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:19.530084 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:19.601401 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:19.623826 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:17:19.626227 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 106258us
I20240501 20:17:19.634878 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:19.642004 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:19.650992 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:19.662988 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:17:19.668766 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:17:19.765769 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:17:19.857349 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:17:19.910667 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:17:19.924950 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:19.950184 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:19.995129 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:20.013420 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:20.015730 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:20.026788 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:20.045797 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:20.069471 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:20.073990 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:20.097143 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:20.097280 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:20.106065 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:20.127118 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:20.162494 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:17:20.176117 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:20.210773 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:20.235621 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:20.238132 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:17:20.394369 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:20.473685 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:20.493484 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:20.524479 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:20.527848 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:20.541616 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:20.544911 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:20.565217 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:20.601626 27460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:17:20.765028 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:20.771660 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:20.782987 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:20.825277 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:20.831477 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:20.842602 27332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 61878.9us
I20240501 20:17:20.883502 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:20.924319 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:20.929338 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:20.981645 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:21.135629 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:21.152477 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:21.176460 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:17:21.178472 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:21.228375 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:21.259228 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:21.280326 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:21.305212 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:21.455700 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:17:21.456732 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:21.477818 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:21.503283 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:21.519526 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:21.552062 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:21.555513 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:21.581856 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:21.605142 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:21.629427 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:17:21.678017 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:17:21.738440 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:17:21.753538 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:21.824287 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:21.855357 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:21.859856 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:21.882606 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:21.888082 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:21.917680 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:21.952561 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:22.007097 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:22.026054 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:22.053546 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:22.073513 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:22.076534 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:22.101936 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:22.101980 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:22.120204 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:22.186400 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:17:22.192416 27200 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:17:22.202436 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:17:22.262183 27460 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:17:22.297700 27200 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:17:22.318151 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:22.327199 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:22.333818 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:22.346760 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:22.354209 27200 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:17:22.358309 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:22.381434 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:22.384166 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:22.388769 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:17:22.408710 27461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 44082.7us
W20240501 20:17:22.515592 27330 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:17:22.516022 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:22.691833 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:17:22.782071 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:22.819444 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:22.823400 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:22.875043 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:22.920156 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:22.968473 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:22.969170 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:22.982748 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:22.990972 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:23.006953 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:23.039239 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:23.067433 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:23.103976 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:23.137936 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:23.194844 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:23.223434 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:23.240180 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:23.245220 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:23.266880 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:23.290474 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:23.310643 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:23.324091 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:23.325326 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:23.325985 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:23.345604 27459 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 51984.9us
I20240501 20:17:23.365984 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:23.368129 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:23.371470 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:23.415941 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:23.434322 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:23.484148 27330 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:17:23.513315 27460 consensus_peers.cc:462] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:23.756645 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:23.763556 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:23.793121 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:23.794525 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:23.812391 27200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:17:23.839560 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:23.855922 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:23.901093 27460 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:17:23.905205 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:23.969910 12344 raft_consensus.cc:998] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.145s)
W20240501 20:17:23.973223 27460 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:23.974382 12344 consensus_queue.cc:237] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:23.983889 29335 raft_consensus.cc:1261] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:23.984653 11857 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:23.991660 11857 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:23.995527 29335 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:23.997397 26993 catalog_manager.cc:5484] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
W20240501 20:17:24.033439 27460 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:17:24.042698 26979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:24.044519 27525 consensus_queue.cc:237] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:24.046849 29335 raft_consensus.cc:1261] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:24.047372 12425 consensus_queue.cc:1035] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:17:24.047673 27461 consensus_peers.cc:485] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00e3777032eb4d9c80f47855e5f9f87a. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:24.049350 26979 catalog_manager.cc:4573] TS d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 00e3777032eb4d9c80f47855e5f9f87a: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:24.050144 11857 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:24.051029 29335 raft_consensus.cc:2932] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:24.054682 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 00e3777032eb4d9c80f47855e5f9f87a with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:24.056874 26985 catalog_manager.cc:5484] T 00e3777032eb4d9c80f47855e5f9f87a P 212de34e3aa24fa1be12c7421df985fc reported cstate change: config changed from index 8 to 9, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
W20240501 20:17:24.066373 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:17:24.142405 27460 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:17:24.169150 27330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:17:24.215541 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:24.253222 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:24.265450 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:24.290520 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:24.316015 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:24.329056 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:24.349916 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:24.357703 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:24.369944 27331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.200:44949, user_credentials={real_user=slave}} blocked reactor thread for 51390.8us
I20240501 20:17:24.392966 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:24.397756 27069 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:17:24.404103 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:24.439289 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:24.454910 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:24.467377 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:24.510128 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:24.548372 12515 ts_tablet_manager.cc:902] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
W20240501 20:17:24.620474 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:17:24.765147 12515 tablet_copy_client.cc:308] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:17:24.789485 12544 raft_consensus.cc:998] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.019s)
I20240501 20:17:24.790549 12544 consensus_queue.cc:237] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:17:24.793077 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 00e3777032eb4d9c80f47855e5f9f87a from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:50389)
I20240501 20:17:24.793259 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 00e3777032eb4d9c80f47855e5f9f87a from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:50389: session id = e957d29835544ac587917cc067fb7423-00e3777032eb4d9c80f47855e5f9f87a
I20240501 20:17:24.797089 27545 tablet_copy_source_session.cc:205] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:24.799382 12515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e3777032eb4d9c80f47855e5f9f87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:24.815366 12515 tablet_copy_client.cc:783] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:24.833721 12515 tablet_copy_client.cc:647] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:24.835944 12515 tablet_copy_client.cc:523] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:24.840965 11857 raft_consensus.cc:998] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.021s)
I20240501 20:17:24.841665 11857 consensus_queue.cc:237] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:24.842895 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:24.846879 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:24.850148 12515 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:17:24.866514 12159 raft_consensus.cc:998] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.226s)
I20240501 20:17:24.867841 12159 consensus_queue.cc:237] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:24.898802 12515 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:24.899387 12515 tablet_bootstrap.cc:492] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:17:24.899843 12515 ts_tablet_manager.cc:1367] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.050s user 0.030s sys 0.000s
I20240501 20:17:24.900998 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:24.900949 12515 raft_consensus.cc:348] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:24.901228 12515 raft_consensus.cc:729] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:17:24.901553 12515 consensus_queue.cc:260] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:24.908944 12515 ts_tablet_manager.cc:1398] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
I20240501 20:17:24.909814 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session e957d29835544ac587917cc067fb7423-00e3777032eb4d9c80f47855e5f9f87a received from {username='slave'} at 127.14.106.200:50389
I20240501 20:17:24.909958 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session e957d29835544ac587917cc067fb7423-00e3777032eb4d9c80f47855e5f9f87a on tablet 00e3777032eb4d9c80f47855e5f9f87a with peer e957d29835544ac587917cc067fb7423
I20240501 20:17:24.922991 12578 raft_consensus.cc:998] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.287s)
I20240501 20:17:24.924170 29068 raft_consensus.cc:1261] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:24.925101 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:24.925040 12558 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:24.927001 12578 consensus_queue.cc:237] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
W20240501 20:17:24.928815 27068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.199:32969, user_credentials={real_user=slave}} blocked reactor thread for 131276us
I20240501 20:17:24.931000 12565 raft_consensus.cc:998] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.212s)
I20240501 20:17:24.931946 12565 consensus_queue.cc:237] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:24.949390 12562 raft_consensus.cc:998] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.017s)
I20240501 20:17:24.950268 12562 consensus_queue.cc:237] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:24.953011 29467 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:24.953822 12499 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:24.961549 29466 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:24.963797 29335 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:24.964578 12499 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:24.965088 27264 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:24.966638 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(00e3777032eb4d9c80f47855e5f9f87a): 4834 bytes on disk
I20240501 20:17:24.966673 12582 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:24.967152 12499 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:24.967281 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(00e3777032eb4d9c80f47855e5f9f87a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240501 20:17:24.976406 27266 raft_consensus.cc:1261] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:24.981825 29069 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:24.981504 12499 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:24.982851 12579 consensus_queue.cc:1035] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:24.988348 12578 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:24.990933 29466 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:24.990419 27265 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:24.997874 26992 catalog_manager.cc:5484] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.011140 11857 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.011205 12544 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.016569 29068 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.017714 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:25.016623 29069 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.018704 12573 raft_consensus.cc:998] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.372s)
I20240501 20:17:25.019335 12573 consensus_queue.cc:237] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:25.027323 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:25.034157 12544 raft_consensus.cc:998] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.308s)
I20240501 20:17:25.034564 27265 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.034982 12544 consensus_queue.cc:237] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:25.035394 12566 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.037621 29334 raft_consensus.cc:1261] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.038261 12570 consensus_queue.cc:1035] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.046602 12566 raft_consensus.cc:998] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.280s)
I20240501 20:17:25.047667 12566 consensus_queue.cc:237] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:25.059803 29466 raft_consensus.cc:1261] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:25.060957 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:25.061043 12558 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:25.061688 12558 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.068189 12581 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.071197 12570 raft_consensus.cc:998] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.095s)
I20240501 20:17:25.070560 29334 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.071887 12570 consensus_queue.cc:237] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:25.075161 12499 raft_consensus.cc:998] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.106s)
I20240501 20:17:25.075311 29336 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:17:25.075776 12499 consensus_queue.cc:237] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:25.077638 12565 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:25.081326 26991 catalog_manager.cc:5484] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.083554 27264 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:17:25.084244 12499 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.091056 27394 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.091780 12570 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.093950 29467 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.095319 11246 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.104220 27135 raft_consensus.cc:1261] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:17:25.104179 29335 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.105037 11435 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:17:25.107017 12566 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.107097 29466 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.110409 12570 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.111058 11435 raft_consensus.cc:998] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.138s)
I20240501 20:17:25.111738 11435 consensus_queue.cc:237] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:17:25.113073 29336 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.118696 11246 raft_consensus.cc:998] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.529s)
W20240501 20:17:25.119084 27069 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:17:25.119727 11246 consensus_queue.cc:237] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:25.122452 29467 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.129024 29465 raft_consensus.cc:1203] T 00e3777032eb4d9c80f47855e5f9f87a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:17:25.132081 27264 raft_consensus.cc:2932] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.133848 27394 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.136250 26993 catalog_manager.cc:5484] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 8 to 9, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 2 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.143652 12573 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.143976 27135 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.148764 12558 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.153698 29465 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.163589 29335 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.164211 11246 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.179627 12565 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.184723 29335 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
W20240501 20:17:25.189669 27200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.198:46843, user_credentials={real_user=slave}} blocked reactor thread for 79388.2us
I20240501 20:17:25.189905 12159 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.191035 27524 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:17:25.192597 29069 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:17:25.199366 12562 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:17:25.200656 27070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:46281, user_credentials={real_user=slave}} blocked reactor thread for 48495.6us
I20240501 20:17:25.201617 27263 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.205806 27265 raft_consensus.cc:1261] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.211599 27265 raft_consensus.cc:2932] T 78fcaa70e916442ea9c67cc762c8db6d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.212544 12617 consensus_queue.cc:1035] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.215430 12562 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:17:25.215965 12573 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.223037 29069 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.228754 27264 raft_consensus.cc:2932] T 5272c34458af43bebb5ba5baefac7137 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.237041 26985 catalog_manager.cc:5484] T 9721fc9afa4741b78823e293514e6265 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.237303 26992 catalog_manager.cc:5484] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.237776 26985 catalog_manager.cc:5484] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.237931 26992 catalog_manager.cc:5484] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.246618 27525 consensus_queue.cc:237] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
W20240501 20:17:25.248708 27461 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Couldn't send request to peer d8b307156e1346bd9eecf655f49924c7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.249240 26987 catalog_manager.cc:5484] T 5272c34458af43bebb5ba5baefac7137 P 212de34e3aa24fa1be12c7421df985fc reported cstate change: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:25.250497 29069 raft_consensus.cc:1261] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:25.251148 12579 consensus_queue.cc:1035] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:25.253860 12579 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.258846 26986 catalog_manager.cc:5484] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.259377 26986 catalog_manager.cc:5484] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 8 to 9, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 2 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.260929 29068 raft_consensus.cc:2932] T 6c836d9afa0f439b978bc2fe5b5e499d P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.267478 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6c836d9afa0f439b978bc2fe5b5e499d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:25.267738 26990 catalog_manager.cc:5484] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 2 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.269704 27525 consensus_queue.cc:237] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:17:25.274132 29467 raft_consensus.cc:1261] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:25.276008 12579 consensus_queue.cc:1035] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:25.279036 12579 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
I20240501 20:17:25.279752 29465 raft_consensus.cc:2932] T 7bb13c1dce8640b28063daa0132d2896 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
W20240501 20:17:25.281919 27461 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Couldn't send request to peer 4d603ea5d17441b3bac562bf08ec83fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bb13c1dce8640b28063daa0132d2896. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.284138 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7bb13c1dce8640b28063daa0132d2896 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:25.309005 27135 consensus_queue.cc:237] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:25.311815 29335 raft_consensus.cc:1261] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:17:25.323014 27525 consensus_queue.cc:237] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:25.327517 12558 raft_consensus.cc:998] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.355s)
W20240501 20:17:25.327700 27462 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Couldn't send request to peer e677127814314770b3370764d0ff9d06. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.328141 12558 consensus_queue.cc:237] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
W20240501 20:17:25.328527 27069 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): Couldn't send request to peer 524f5e87a3ce40378bde76e667a4ece7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.330561 29068 raft_consensus.cc:1261] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:25.331524 12579 consensus_queue.cc:1035] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:25.335768 12579 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.339247 12581 consensus_queue.cc:1035] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:17:25.339210 29068 raft_consensus.cc:2932] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.340378 27394 consensus_queue.cc:237] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:17:25.340461 26993 catalog_manager.cc:5484] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 6 to 7, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
W20240501 20:17:25.347101 27332 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17e69dbd245e4b8db397aaffc9145c9f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.349210 26985 catalog_manager.cc:5484] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 8 to 9, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } }
I20240501 20:17:25.349742 26985 catalog_manager.cc:5484] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 8 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.352643 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0c836bb145a54160966c475e1490da38 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:25.353178 12581 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:25.355793 27263 raft_consensus.cc:1261] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:17:25.356504 12159 consensus_queue.cc:1035] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:25.357549 27264 consensus_queue.cc:237] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:25.359053 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e7f03e02d13f44d188e5eb2774771a0d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:25.359592 27265 consensus_queue.cc:237] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:25.359651 29334 raft_consensus.cc:2932] T e7f03e02d13f44d188e5eb2774771a0d P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:25.360713 12159 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.363371 27266 raft_consensus.cc:2932] T 17e69dbd245e4b8db397aaffc9145c9f P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.372099 26981 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 17e69dbd245e4b8db397aaffc9145c9f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 20:17:25.376740 26979 catalog_manager.cc:4573] TS d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9721fc9afa4741b78823e293514e6265: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:25.383076 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): Couldn't send request to peer d56cbcb06b9e418cb617de6a34b946b0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:17:25.394464 29068 raft_consensus.cc:1261] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:17:25.394213 12562 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:25.397502 11435 consensus_queue.cc:1035] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:17:25.400180 27394 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:17:25.400750 27135 raft_consensus.cc:1261] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.400485 27524 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:25.400982 12558 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:17:25.403438 11435 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.403646 26990 catalog_manager.cc:5484] T 0c836bb145a54160966c475e1490da38 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 8 to 9, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.404312 11435 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.410823 29068 raft_consensus.cc:2932] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.417980 26987 catalog_manager.cc:5484] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 9 to 10, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New cstate: current_term: 2 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
W20240501 20:17:25.419752 27201 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5b9f1a087114c82b010bd10e4f676c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.421845 26986 catalog_manager.cc:5484] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 7 to 8, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 2 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:25.422894 12562 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.424448 26991 catalog_manager.cc:5484] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:25.429988 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 74c414c2eed045bda9d85fe5c2fea654 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:25.432222 27135 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.434886 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e5b9f1a087114c82b010bd10e4f676c1 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:25.435623 12558 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:25.438236 27394 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:25.447407 26987 catalog_manager.cc:5484] T 74c414c2eed045bda9d85fe5c2fea654 P 565cade0058540a4aa18d7c7a25affbe reported cstate change: config changed from index 9 to 10, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New cstate: current_term: 2 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:25.451359 26990 catalog_manager.cc:5484] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.453150 26992 catalog_manager.cc:5484] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 5 to 6, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
W20240501 20:17:25.456712 27202 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Couldn't send request to peer e677127814314770b3370764d0ff9d06. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74c414c2eed045bda9d85fe5c2fea654. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.463696 27265 consensus_queue.cc:237] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
W20240501 20:17:25.467614 27201 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 -> Peer e957d29835544ac587917cc067fb7423 (127.14.106.200:44949): Couldn't send request to peer e957d29835544ac587917cc067fb7423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26cb2329d176441ca9127187db2d157a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.468631 27524 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.470953 12573 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.473946 12573 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.480393 27524 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.485949 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 26cb2329d176441ca9127187db2d157a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:25.490520 26988 catalog_manager.cc:5484] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 6 to 7, VOTER e957d29835544ac587917cc067fb7423 (127.14.106.200) added. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:25.493986 27525 consensus_queue.cc:237] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
W20240501 20:17:25.497583 27461 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Couldn't send request to peer d8b307156e1346bd9eecf655f49924c7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:25.500252 27394 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:17:25.502086 12579 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:17:25.504918 12579 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.510216 27395 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.518690 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 944d354124664bc0870f251d8a0831c5 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:25.518431 26989 catalog_manager.cc:5484] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 6 to 7, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) added. New cstate: current_term: 1 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:25.578573 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:25.599323 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:25.641083 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:25.688140 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:25.718026 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:25.727474 12681 ts_tablet_manager.cc:902] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:17:25.758492 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:25.763716 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:25.777848 12683 ts_tablet_manager.cc:902] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:17:25.841123 12681 tablet_copy_client.cc:308] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:17:25.842159 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 6c836d9afa0f439b978bc2fe5b5e499d from peer d8b307156e1346bd9eecf655f49924c7 ({username='slave'} at 127.14.106.194:53599)
I20240501 20:17:25.842330 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 6c836d9afa0f439b978bc2fe5b5e499d from peer d8b307156e1346bd9eecf655f49924c7 at {username='slave'} at 127.14.106.194:53599: session id = d8b307156e1346bd9eecf655f49924c7-6c836d9afa0f439b978bc2fe5b5e499d
I20240501 20:17:25.843641 12698 ts_tablet_manager.cc:902] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:17:25.846926 27545 tablet_copy_source_session.cc:205] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:25.852406 12683 tablet_copy_client.cc:308] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:17:25.857339 12681 ts_tablet_manager.cc:1875] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:17:25.862607 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 0c836bb145a54160966c475e1490da38 from peer e677127814314770b3370764d0ff9d06 ({username='slave'} at 127.14.106.195:33343)
I20240501 20:17:25.862907 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 0c836bb145a54160966c475e1490da38 from peer e677127814314770b3370764d0ff9d06 at {username='slave'} at 127.14.106.195:33343: session id = e677127814314770b3370764d0ff9d06-0c836bb145a54160966c475e1490da38
I20240501 20:17:25.863278 12678 ts_tablet_manager.cc:902] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:17:25.870095 27545 tablet_copy_source_session.cc:205] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:25.872496 12683 ts_tablet_manager.cc:1875] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:17:25.887177 12683 ts_tablet_manager.cc:1888] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:17:25.928772 12713 ts_tablet_manager.cc:902] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:17:25.931701 12683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c836bb145a54160966c475e1490da38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:25.956311 12724 ts_tablet_manager.cc:902] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: Initiating tablet copy from peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:17:25.957185 12720 ts_tablet_manager.cc:902] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:17:25.983321 12733 ts_tablet_manager.cc:902] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:17:26.012645 12733 tablet_copy_client.cc:308] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:17:26.014685 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:26.014967 12681 ts_tablet_manager.cc:1888] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 20:17:26.015421 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c836d9afa0f439b978bc2fe5b5e499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:26.017648 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:26.018093 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:26.042762 12683 tablet_copy_client.cc:783] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.050930 12683 tablet_copy_client.cc:647] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.053690 12698 tablet_copy_client.cc:308] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:17:26.054587 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet e7f03e02d13f44d188e5eb2774771a0d from peer 524f5e87a3ce40378bde76e667a4ece7 ({username='slave'} at 127.14.106.196:52025)
I20240501 20:17:26.054782 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet e7f03e02d13f44d188e5eb2774771a0d from peer 524f5e87a3ce40378bde76e667a4ece7 at {username='slave'} at 127.14.106.196:52025: session id = 524f5e87a3ce40378bde76e667a4ece7-e7f03e02d13f44d188e5eb2774771a0d
I20240501 20:17:26.060688 27154 tablet_copy_source_session.cc:205] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:26.062429 12713 tablet_copy_client.cc:308] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:46281
I20240501 20:17:26.066183 12698 ts_tablet_manager.cc:1875] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:17:26.067117 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:26.068984 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:26.083884 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet 17e69dbd245e4b8db397aaffc9145c9f from peer 565cade0058540a4aa18d7c7a25affbe ({username='slave'} at 127.14.106.197:44727)
I20240501 20:17:26.084066 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet 17e69dbd245e4b8db397aaffc9145c9f from peer 565cade0058540a4aa18d7c7a25affbe at {username='slave'} at 127.14.106.197:44727: session id = 565cade0058540a4aa18d7c7a25affbe-17e69dbd245e4b8db397aaffc9145c9f
I20240501 20:17:26.087873 27416 tablet_copy_source_session.cc:205] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:26.090024 12733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e69dbd245e4b8db397aaffc9145c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:26.112362 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:26.116063 12733 tablet_copy_client.cc:783] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.119558 12683 tablet_copy_client.cc:523] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.129384 12733 tablet_copy_client.cc:647] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.130389 12678 tablet_copy_client.cc:308] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:17:26.134109 12681 tablet_copy_client.cc:783] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.140240 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 7bb13c1dce8640b28063daa0132d2896 from peer 4d603ea5d17441b3bac562bf08ec83fe ({username='slave'} at 127.14.106.193:56005)
I20240501 20:17:26.140404 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 7bb13c1dce8640b28063daa0132d2896 from peer 4d603ea5d17441b3bac562bf08ec83fe at {username='slave'} at 127.14.106.193:56005: session id = 4d603ea5d17441b3bac562bf08ec83fe-7bb13c1dce8640b28063daa0132d2896
I20240501 20:17:26.141300 12768 ts_tablet_manager.cc:902] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: Initiating tablet copy from peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:17:26.145890 27545 tablet_copy_source_session.cc:205] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:26.146034 12698 ts_tablet_manager.cc:1888] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 20:17:26.146306 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f03e02d13f44d188e5eb2774771a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:26.146986 27286 tablet_copy_service.cc:140] P d8b307156e1346bd9eecf655f49924c7: Received BeginTabletCopySession request for tablet e5b9f1a087114c82b010bd10e4f676c1 from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:39693)
I20240501 20:17:26.147127 27286 tablet_copy_service.cc:161] P d8b307156e1346bd9eecf655f49924c7: Beginning new tablet copy session on tablet e5b9f1a087114c82b010bd10e4f676c1 from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:39693: session id = e957d29835544ac587917cc067fb7423-e5b9f1a087114c82b010bd10e4f676c1
I20240501 20:17:26.148025 12678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb13c1dce8640b28063daa0132d2896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:26.150754 27286 tablet_copy_source_session.cc:205] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:26.152199 12713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b9f1a087114c82b010bd10e4f676c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:26.158780 12720 tablet_copy_client.cc:308] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:17:26.159382 12713 tablet_copy_client.cc:783] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.159372 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 944d354124664bc0870f251d8a0831c5 from peer d8b307156e1346bd9eecf655f49924c7 ({username='slave'} at 127.14.106.194:53599)
I20240501 20:17:26.159495 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 944d354124664bc0870f251d8a0831c5 from peer d8b307156e1346bd9eecf655f49924c7 at {username='slave'} at 127.14.106.194:53599: session id = d8b307156e1346bd9eecf655f49924c7-944d354124664bc0870f251d8a0831c5
I20240501 20:17:26.160130 12724 tablet_copy_client.cc:308] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:46281
I20240501 20:17:26.160770 27285 tablet_copy_service.cc:140] P d8b307156e1346bd9eecf655f49924c7: Received BeginTabletCopySession request for tablet 74c414c2eed045bda9d85fe5c2fea654 from peer e677127814314770b3370764d0ff9d06 ({username='slave'} at 127.14.106.195:45589)
I20240501 20:17:26.160892 27285 tablet_copy_service.cc:161] P d8b307156e1346bd9eecf655f49924c7: Beginning new tablet copy session on tablet 74c414c2eed045bda9d85fe5c2fea654 from peer e677127814314770b3370764d0ff9d06 at {username='slave'} at 127.14.106.195:45589: session id = e677127814314770b3370764d0ff9d06-74c414c2eed045bda9d85fe5c2fea654
I20240501 20:17:26.162766 27545 tablet_copy_source_session.cc:205] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:26.163735 12720 ts_tablet_manager.cc:1875] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:17:26.164000 27285 tablet_copy_source_session.cc:205] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:26.165515 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c414c2eed045bda9d85fe5c2fea654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:26.171280 12733 tablet_copy_client.cc:523] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.173200 12698 tablet_copy_client.cc:783] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.173794 12713 tablet_copy_client.cc:647] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.175551 12678 tablet_copy_client.cc:783] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.178983 12713 tablet_copy_client.cc:523] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.181273 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:26.202971 12733 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: Bootstrap starting.
I20240501 20:17:26.203693 12713 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:17:26.208514 12678 tablet_copy_client.cc:647] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.209697 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:26.211232 12678 tablet_copy_client.cc:523] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.212635 12724 tablet_copy_client.cc:783] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.216595 12678 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap starting.
I20240501 20:17:26.220765 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:26.222792 12724 tablet_copy_client.cc:647] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.226681 12724 tablet_copy_client.cc:523] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.243336 12683 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:17:26.256508 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:26.258391 12681 tablet_copy_client.cc:647] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.266877 12698 tablet_copy_client.cc:647] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.273366 12698 tablet_copy_client.cc:523] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.277525 12681 tablet_copy_client.cc:523] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.282756 12681 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:17:26.288193 12698 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:17:26.305114 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:26.336854 12720 ts_tablet_manager.cc:1888] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 20:17:26.337250 12720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d354124664bc0870f251d8a0831c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:26.344221 12724 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:17:26.345027 12713 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.358170 12713 tablet_bootstrap.cc:492] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:17:26.358476 12713 ts_tablet_manager.cc:1367] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.155s user 0.020s sys 0.017s
I20240501 20:17:26.360798 12713 raft_consensus.cc:348] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:26.361004 12713 raft_consensus.cc:729] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.363174 12713 consensus_queue.cc:260] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:26.386979 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:26.416443 12681 tablet_bootstrap.cc:700] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Time spent opening tablet: real 0.132s user 0.002s sys 0.000s
I20240501 20:17:26.428611 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:17:26.429636 26979 catalog_manager.cc:4573] TS d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 00e3777032eb4d9c80f47855e5f9f87a: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:26.443517 12768 tablet_copy_client.cc:308] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:46281
I20240501 20:17:26.444293 27285 tablet_copy_service.cc:140] P d8b307156e1346bd9eecf655f49924c7: Received BeginTabletCopySession request for tablet 26cb2329d176441ca9127187db2d157a from peer e957d29835544ac587917cc067fb7423 ({username='slave'} at 127.14.106.200:39693)
I20240501 20:17:26.444439 27285 tablet_copy_service.cc:161] P d8b307156e1346bd9eecf655f49924c7: Beginning new tablet copy session on tablet 26cb2329d176441ca9127187db2d157a from peer e957d29835544ac587917cc067fb7423 at {username='slave'} at 127.14.106.200:39693: session id = e957d29835544ac587917cc067fb7423-26cb2329d176441ca9127187db2d157a
I20240501 20:17:26.448004 27285 tablet_copy_source_session.cc:205] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:26.449512 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cb2329d176441ca9127187db2d157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:26.455325 12720 tablet_copy_client.cc:783] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.457082 12768 tablet_copy_client.cc:783] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:26.463310 12768 tablet_copy_client.cc:647] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.466199 12768 tablet_copy_client.cc:523] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.467298 12720 tablet_copy_client.cc:647] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:26.470000 12768 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: Bootstrap starting.
I20240501 20:17:26.473292 12713 ts_tablet_manager.cc:1398] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.114s user 0.003s sys 0.038s
I20240501 20:17:26.473923 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:26.474109 27285 tablet_copy_service.cc:342] P d8b307156e1346bd9eecf655f49924c7: Request end of tablet copy session e957d29835544ac587917cc067fb7423-e5b9f1a087114c82b010bd10e4f676c1 received from {username='slave'} at 127.14.106.200:39693
I20240501 20:17:26.474224 27285 tablet_copy_service.cc:433] P d8b307156e1346bd9eecf655f49924c7: ending tablet copy session e957d29835544ac587917cc067fb7423-e5b9f1a087114c82b010bd10e4f676c1 on tablet e5b9f1a087114c82b010bd10e4f676c1 with peer e957d29835544ac587917cc067fb7423
I20240501 20:17:26.476181 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(e5b9f1a087114c82b010bd10e4f676c1): 4834 bytes on disk
I20240501 20:17:26.476723 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(e5b9f1a087114c82b010bd10e4f676c1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.547977 12733 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.548409 12733 tablet_bootstrap.cc:492] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: Bootstrap complete.
I20240501 20:17:26.548727 12733 ts_tablet_manager.cc:1367] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: Time spent bootstrapping tablet: real 0.346s user 0.024s sys 0.000s
I20240501 20:17:26.549520 12733 raft_consensus.cc:348] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:17:26.549767 12733 raft_consensus.cc:729] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565cade0058540a4aa18d7c7a25affbe, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.550019 12733 consensus_queue.cc:260] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:17:26.553997 12768 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.554446 12768 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: Bootstrap complete.
I20240501 20:17:26.557814 12768 ts_tablet_manager.cc:1367] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: Time spent bootstrapping tablet: real 0.088s user 0.021s sys 0.012s
I20240501 20:17:26.558701 12768 raft_consensus.cc:348] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:26.558924 12768 raft_consensus.cc:729] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957d29835544ac587917cc067fb7423, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.559723 12768 consensus_queue.cc:260] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:26.560362 12683 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.568156 12768 ts_tablet_manager.cc:1398] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:17:26.568861 27285 tablet_copy_service.cc:342] P d8b307156e1346bd9eecf655f49924c7: Request end of tablet copy session e957d29835544ac587917cc067fb7423-26cb2329d176441ca9127187db2d157a received from {username='slave'} at 127.14.106.200:39693
I20240501 20:17:26.568989 27285 tablet_copy_service.cc:433] P d8b307156e1346bd9eecf655f49924c7: ending tablet copy session e957d29835544ac587917cc067fb7423-26cb2329d176441ca9127187db2d157a on tablet 26cb2329d176441ca9127187db2d157a with peer e957d29835544ac587917cc067fb7423
I20240501 20:17:26.569957 12683 tablet_bootstrap.cc:492] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Bootstrap complete.
I20240501 20:17:26.570288 12683 ts_tablet_manager.cc:1367] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.327s user 0.018s sys 0.016s
I20240501 20:17:26.572486 12683 raft_consensus.cc:348] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:26.573124 12683 raft_consensus.cc:729] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.573832 12683 consensus_queue.cc:260] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:26.574616 12683 ts_tablet_manager.cc:1398] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:17:26.575338 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session e677127814314770b3370764d0ff9d06-0c836bb145a54160966c475e1490da38 received from {username='slave'} at 127.14.106.195:33343
I20240501 20:17:26.575453 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session e677127814314770b3370764d0ff9d06-0c836bb145a54160966c475e1490da38 on tablet 0c836bb145a54160966c475e1490da38 with peer e677127814314770b3370764d0ff9d06
I20240501 20:17:26.583209 12724 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.583635 12724 tablet_bootstrap.cc:492] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: Bootstrap complete.
I20240501 20:17:26.583925 12724 ts_tablet_manager.cc:1367] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.266s user 0.017s sys 0.008s
I20240501 20:17:26.584873 29465 raft_consensus.cc:1203] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:17:26.586393 12733 ts_tablet_manager.cc:1398] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20240501 20:17:26.590193 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session 565cade0058540a4aa18d7c7a25affbe-17e69dbd245e4b8db397aaffc9145c9f received from {username='slave'} at 127.14.106.197:44727
I20240501 20:17:26.590315 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session 565cade0058540a4aa18d7c7a25affbe-17e69dbd245e4b8db397aaffc9145c9f on tablet 17e69dbd245e4b8db397aaffc9145c9f with peer 565cade0058540a4aa18d7c7a25affbe
I20240501 20:17:26.595918 12724 raft_consensus.cc:348] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:26.596364 12724 raft_consensus.cc:729] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.596792 12724 consensus_queue.cc:260] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:26.600952 12698 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.603212 12724 ts_tablet_manager.cc:1398] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.019s user 0.001s sys 0.002s
I20240501 20:17:26.603472 12698 tablet_bootstrap.cc:492] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap complete.
W20240501 20:17:26.603497 27200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.196:43861, user_credentials={real_user=slave}} blocked reactor thread for 81375.5us
I20240501 20:17:26.603811 12698 ts_tablet_manager.cc:1367] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.325s user 0.027s sys 0.004s
I20240501 20:17:26.605413 12720 tablet_copy_client.cc:523] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:26.606554 12698 raft_consensus.cc:348] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:26.606818 12698 raft_consensus.cc:729] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.607054 12698 consensus_queue.cc:260] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:26.607643 12698 ts_tablet_manager.cc:1398] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:17:26.610146 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 524f5e87a3ce40378bde76e667a4ece7-e7f03e02d13f44d188e5eb2774771a0d received from {username='slave'} at 127.14.106.196:52025
I20240501 20:17:26.610294 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 524f5e87a3ce40378bde76e667a4ece7-e7f03e02d13f44d188e5eb2774771a0d on tablet e7f03e02d13f44d188e5eb2774771a0d with peer 524f5e87a3ce40378bde76e667a4ece7
I20240501 20:17:26.610805 27285 tablet_copy_service.cc:342] P d8b307156e1346bd9eecf655f49924c7: Request end of tablet copy session e677127814314770b3370764d0ff9d06-74c414c2eed045bda9d85fe5c2fea654 received from {username='slave'} at 127.14.106.195:45589
I20240501 20:17:26.610960 27285 tablet_copy_service.cc:433] P d8b307156e1346bd9eecf655f49924c7: ending tablet copy session e677127814314770b3370764d0ff9d06-74c414c2eed045bda9d85fe5c2fea654 on tablet 74c414c2eed045bda9d85fe5c2fea654 with peer e677127814314770b3370764d0ff9d06
I20240501 20:17:26.612003 12720 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Bootstrap starting.
I20240501 20:17:26.633209 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(e7f03e02d13f44d188e5eb2774771a0d): 4581 bytes on disk
I20240501 20:17:26.634040 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(e7f03e02d13f44d188e5eb2774771a0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.665637 12678 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.666267 12678 tablet_bootstrap.cc:492] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: Bootstrap complete.
I20240501 20:17:26.666698 12678 ts_tablet_manager.cc:1367] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent bootstrapping tablet: real 0.450s user 0.025s sys 0.001s
I20240501 20:17:26.667766 12678 raft_consensus.cc:348] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:17:26.678961 12678 raft_consensus.cc:729] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d603ea5d17441b3bac562bf08ec83fe, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.679358 12678 consensus_queue.cc:260] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } }
I20240501 20:17:26.681174 12678 ts_tablet_manager.cc:1398] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:17:26.681876 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session 4d603ea5d17441b3bac562bf08ec83fe-7bb13c1dce8640b28063daa0132d2896 received from {username='slave'} at 127.14.106.193:56005
I20240501 20:17:26.681988 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session 4d603ea5d17441b3bac562bf08ec83fe-7bb13c1dce8640b28063daa0132d2896 on tablet 7bb13c1dce8640b28063daa0132d2896 with peer 4d603ea5d17441b3bac562bf08ec83fe
I20240501 20:17:26.716774 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(0c836bb145a54160966c475e1490da38): 4921 bytes on disk
I20240501 20:17:26.717433 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(0c836bb145a54160966c475e1490da38) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.718431 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(74c414c2eed045bda9d85fe5c2fea654): 4921 bytes on disk
I20240501 20:17:26.718981 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(74c414c2eed045bda9d85fe5c2fea654) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.729025 29513 maintenance_manager.cc:419] P e957d29835544ac587917cc067fb7423: Scheduling UndoDeltaBlockGCOp(26cb2329d176441ca9127187db2d157a): 4581 bytes on disk
I20240501 20:17:26.729578 29404 maintenance_manager.cc:643] P e957d29835544ac587917cc067fb7423: UndoDeltaBlockGCOp(26cb2329d176441ca9127187db2d157a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.744644 27525 raft_consensus.cc:1203] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 20:17:26.747298 12681 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.752496 12681 tablet_bootstrap.cc:492] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Bootstrap complete.
I20240501 20:17:26.752827 12681 ts_tablet_manager.cc:1367] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.470s user 0.025s sys 0.001s
I20240501 20:17:26.761595 12720 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:26.762048 12720 tablet_bootstrap.cc:492] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Bootstrap complete.
I20240501 20:17:26.762378 12720 ts_tablet_manager.cc:1367] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Time spent bootstrapping tablet: real 0.150s user 0.022s sys 0.000s
I20240501 20:17:26.763195 12681 raft_consensus.cc:348] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:26.763422 12681 raft_consensus.cc:729] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.763685 12681 consensus_queue.cc:260] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:26.765717 29116 maintenance_manager.cc:419] P 565cade0058540a4aa18d7c7a25affbe: Scheduling UndoDeltaBlockGCOp(17e69dbd245e4b8db397aaffc9145c9f): 4834 bytes on disk
I20240501 20:17:26.766963 27181 maintenance_manager.cc:419] P 4d603ea5d17441b3bac562bf08ec83fe: Scheduling UndoDeltaBlockGCOp(7bb13c1dce8640b28063daa0132d2896): 4581 bytes on disk
I20240501 20:17:26.767588 29007 maintenance_manager.cc:643] P 565cade0058540a4aa18d7c7a25affbe: UndoDeltaBlockGCOp(17e69dbd245e4b8db397aaffc9145c9f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.767683 27072 maintenance_manager.cc:643] P 4d603ea5d17441b3bac562bf08ec83fe: UndoDeltaBlockGCOp(7bb13c1dce8640b28063daa0132d2896) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.776474 27395 consensus_queue.cc:237] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:26.779192 12681 ts_tablet_manager.cc:1398] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240501 20:17:26.780086 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session d8b307156e1346bd9eecf655f49924c7-6c836d9afa0f439b978bc2fe5b5e499d received from {username='slave'} at 127.14.106.194:53599
I20240501 20:17:26.780201 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session d8b307156e1346bd9eecf655f49924c7-6c836d9afa0f439b978bc2fe5b5e499d on tablet 6c836d9afa0f439b978bc2fe5b5e499d with peer d8b307156e1346bd9eecf655f49924c7
I20240501 20:17:26.779052 12720 raft_consensus.cc:348] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:26.781905 12720 raft_consensus.cc:729] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Initialized, Role: FOLLOWER
I20240501 20:17:26.782157 12720 consensus_queue.cc:260] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:26.782733 12720 ts_tablet_manager.cc:1398] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 20:17:26.783341 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session d8b307156e1346bd9eecf655f49924c7-944d354124664bc0870f251d8a0831c5 received from {username='slave'} at 127.14.106.194:53599
I20240501 20:17:26.783459 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session d8b307156e1346bd9eecf655f49924c7-944d354124664bc0870f251d8a0831c5 on tablet 944d354124664bc0870f251d8a0831c5 with peer d8b307156e1346bd9eecf655f49924c7
I20240501 20:17:26.794163 29465 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:26.794879 29335 raft_consensus.cc:1261] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:26.795877 12159 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:26.796437 12159 consensus_queue.cc:1035] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:26.816946 12819 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:26.820123 29467 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:26.832504 26989 catalog_manager.cc:5484] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 8 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:26.835206 29334 raft_consensus.cc:2932] T 57b4bd50bc8f4d50957bd0280224c420 P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:26.836632 27135 raft_consensus.cc:1203] T 7bb13c1dce8640b28063daa0132d2896 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:17:26.851642 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b4bd50bc8f4d50957bd0280224c420 with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59150
I20240501 20:17:26.852275 12843 tablet_replica.cc:330] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:17:26.852599 12843 raft_consensus.cc:2229] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:17:26.852939 12843 raft_consensus.cc:2258] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:17:26.854081 12843 ts_tablet_manager.cc:1875] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:17:26.870179 12843 ts_tablet_manager.cc:1888] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:17:26.870335 12843 log.cc:1196] T 57b4bd50bc8f4d50957bd0280224c420 P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/57b4bd50bc8f4d50957bd0280224c420
I20240501 20:17:26.871228 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 57b4bd50bc8f4d50957bd0280224c420 (table rebalance_test_table_9 [id=2c64f22861534f1d84ad1e2a9970ff4a]) successfully deleted
I20240501 20:17:26.881808 29068 raft_consensus.cc:1203] T 17e69dbd245e4b8db397aaffc9145c9f P 565cade0058540a4aa18d7c7a25affbe [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:17:26.928555 12819 raft_consensus.cc:998] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06: Attempting to remove follower d56cbcb06b9e418cb617de6a34b946b0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d56cbcb06b9e418cb617de6a34b946b0 for more than 300 seconds (300.458s)
I20240501 20:17:26.929208 12819 consensus_queue.cc:237] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:26.930538 27395 raft_consensus.cc:1203] T 0c836bb145a54160966c475e1490da38 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:17:26.947721 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:26.955533 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(944d354124664bc0870f251d8a0831c5): 4834 bytes on disk
I20240501 20:17:26.956262 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(944d354124664bc0870f251d8a0831c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.958132 27312 maintenance_manager.cc:419] P d8b307156e1346bd9eecf655f49924c7: Scheduling UndoDeltaBlockGCOp(6c836d9afa0f439b978bc2fe5b5e499d): 4581 bytes on disk
I20240501 20:17:26.958875 27203 maintenance_manager.cc:643] P d8b307156e1346bd9eecf655f49924c7: UndoDeltaBlockGCOp(6c836d9afa0f439b978bc2fe5b5e499d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240501 20:17:26.959995 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:26.979887 29334 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:26.980561 12159 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:26.983997 12159 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:26.989856 29334 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:26.991080 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:26.994875 27395 raft_consensus.cc:1203] T 74c414c2eed045bda9d85fe5c2fea654 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 20:17:27.002888 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:27.007133 29465 raft_consensus.cc:1203] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:17:27.012116 26988 catalog_manager.cc:5484] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc reported cstate change: config changed from index 7 to 8, VOTER d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198) evicted. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:27.019157 27395 consensus_queue.cc:237] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
W20240501 20:17:27.022080 27330 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): Couldn't send request to peer 524f5e87a3ce40378bde76e667a4ece7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:27.023896 29334 raft_consensus.cc:1261] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:27.024492 12159 consensus_queue.cc:1035] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:27.026747 12819 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:27.031204 29334 raft_consensus.cc:2932] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:27.034493 26981 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c619a2235c9248eeb6b2e1d78583d1ba with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:27.040383 26988 catalog_manager.cc:5484] T c619a2235c9248eeb6b2e1d78583d1ba P 212de34e3aa24fa1be12c7421df985fc reported cstate change: config changed from index 8 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:27.046882 27265 raft_consensus.cc:1203] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:17:27.087185 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:27.121871 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:27.134544 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:27.343538 27265 raft_consensus.cc:1203] T 6c836d9afa0f439b978bc2fe5b5e499d P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:17:27.396018 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:27.403942 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:27.420608 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:27.503242 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:27.524694 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:27.538940 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:27.539669 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 20:17:27.590073 26979 catalog_manager.cc:4573] TS d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 57b4bd50bc8f4d50957bd0280224c420: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:27.599478 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:27.604367 12968 ts_tablet_manager.cc:902] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
I20240501 20:17:27.634119 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:27.657457 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:27.719558 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:27.739065 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:27.785271 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:27.794450 12968 tablet_copy_client.cc:308] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:35443
I20240501 20:17:27.795351 27416 tablet_copy_service.cc:140] P e677127814314770b3370764d0ff9d06: Received BeginTabletCopySession request for tablet c619a2235c9248eeb6b2e1d78583d1ba from peer 524f5e87a3ce40378bde76e667a4ece7 ({username='slave'} at 127.14.106.196:53879)
I20240501 20:17:27.795534 27416 tablet_copy_service.cc:161] P e677127814314770b3370764d0ff9d06: Beginning new tablet copy session on tablet c619a2235c9248eeb6b2e1d78583d1ba from peer 524f5e87a3ce40378bde76e667a4ece7 at {username='slave'} at 127.14.106.196:53879: session id = 524f5e87a3ce40378bde76e667a4ece7-c619a2235c9248eeb6b2e1d78583d1ba
I20240501 20:17:27.796746 27135 consensus_queue.cc:237] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:27.801527 27416 tablet_copy_source_session.cc:205] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:27.803210 12968 ts_tablet_manager.cc:1875] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:17:27.807015 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:27.811594 12968 ts_tablet_manager.cc:1888] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:17:27.811955 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c619a2235c9248eeb6b2e1d78583d1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:27.818542 12968 tablet_copy_client.cc:783] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:27.827443 29465 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:27.827679 12968 tablet_copy_client.cc:647] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:27.828065 12566 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:27.832814 12968 tablet_copy_client.cc:523] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:27.839062 12566 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:27.843550 29466 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:27.850069 26988 catalog_manager.cc:5484] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 7 to 8, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:27.858449 12968 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:17:27.935920 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:27.938968 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:27.965665 27135 consensus_queue.cc:237] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:27.967253 27376 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf74c24f28546bb90e10002270e3814 with delete_type TABLET_DATA_TOMBSTONED (TS e677127814314770b3370764d0ff9d06 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:32800
W20240501 20:17:27.968526 27069 consensus_peers.cc:485] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): Couldn't send request to peer 524f5e87a3ce40378bde76e667a4ece7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bf74c24f28546bb90e10002270e3814. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:27.970412 29465 raft_consensus.cc:1261] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer 4d603ea5d17441b3bac562bf08ec83fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:27.971045 12566 consensus_queue.cc:1035] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:27.974035 12566 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:27.976691 13008 tablet_replica.cc:330] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: stopping tablet replica
I20240501 20:17:27.976936 13008 raft_consensus.cc:2229] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:17:27.977216 13008 raft_consensus.cc:2258] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:17:27.977157 29465 raft_consensus.cc:2932] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:27.983907 13008 ts_tablet_manager.cc:1875] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:17:27.988869 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8bf74c24f28546bb90e10002270e3814 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:27.991143 26988 catalog_manager.cc:5484] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe reported cstate change: config changed from index 8 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "4d603ea5d17441b3bac562bf08ec83fe" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:28.001176 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:28.008384 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:28.015648 13008 ts_tablet_manager.cc:1888] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 20:17:28.015813 13008 log.cc:1196] T 8bf74c24f28546bb90e10002270e3814 P e677127814314770b3370764d0ff9d06: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/8bf74c24f28546bb90e10002270e3814
I20240501 20:17:28.016693 26981 catalog_manager.cc:4830] TS e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): tablet 8bf74c24f28546bb90e10002270e3814 (table rebalance_test_table_5 [id=cb0ab861e4ae485c8f71dd1c7e9f1022]) successfully deleted
I20240501 20:17:28.018914 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:28.035451 12968 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:28.036135 12968 tablet_bootstrap.cc:492] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap complete.
I20240501 20:17:28.036484 12968 ts_tablet_manager.cc:1367] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.178s user 0.028s sys 0.010s
I20240501 20:17:28.037317 12968 raft_consensus.cc:348] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:28.037648 12968 raft_consensus.cc:729] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:17:28.037995 12968 consensus_queue.cc:260] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:28.038987 12968 ts_tablet_manager.cc:1398] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:17:28.042145 27416 tablet_copy_service.cc:342] P e677127814314770b3370764d0ff9d06: Request end of tablet copy session 524f5e87a3ce40378bde76e667a4ece7-c619a2235c9248eeb6b2e1d78583d1ba received from {username='slave'} at 127.14.106.196:53879
I20240501 20:17:28.042325 27416 tablet_copy_service.cc:433] P e677127814314770b3370764d0ff9d06: ending tablet copy session 524f5e87a3ce40378bde76e667a4ece7-c619a2235c9248eeb6b2e1d78583d1ba on tablet c619a2235c9248eeb6b2e1d78583d1ba with peer 524f5e87a3ce40378bde76e667a4ece7
I20240501 20:17:28.068073 27376 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:28.084128 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:28.141916 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(c619a2235c9248eeb6b2e1d78583d1ba): 4921 bytes on disk
I20240501 20:17:28.142577 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(c619a2235c9248eeb6b2e1d78583d1ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240501 20:17:28.162272 27525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "944d354124664bc0870f251d8a0831c5"
dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50332
I20240501 20:17:28.162600 27525 raft_consensus.cc:595] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:17:28.185948 27525 raft_consensus.cc:1203] T c619a2235c9248eeb6b2e1d78583d1ba P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:17:28.265061 27265 consensus_queue.cc:237] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:28.282848 29465 raft_consensus.cc:1261] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:17:28.283408 12573 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:28.319416 29465 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:28.328869 26988 catalog_manager.cc:5484] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 7 to 8, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:28.331943 12579 raft_consensus.cc:979] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: : Instructing follower e677127814314770b3370764d0ff9d06 to start an election
I20240501 20:17:28.332082 12579 raft_consensus.cc:1067] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Signalling peer e677127814314770b3370764d0ff9d06 to start an election
I20240501 20:17:28.332732 27395 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "944d354124664bc0870f251d8a0831c5"
dest_uuid: "e677127814314770b3370764d0ff9d06"
from {username='slave'} at 127.14.106.196:53879
I20240501 20:17:28.332948 27395 raft_consensus.cc:482] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:17:28.333063 27395 raft_consensus.cc:3037] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:17:28.334987 27113 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b9f1a087114c82b010bd10e4f676c1 with delete_type TABLET_DATA_TOMBSTONED (TS 4d603ea5d17441b3bac562bf08ec83fe not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41322
I20240501 20:17:28.336292 27395 raft_consensus.cc:504] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:28.339968 27395 leader_election.cc:288] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 2 election: Requested vote from peers 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:17:28.339918 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d354124664bc0870f251d8a0831c5" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
I20240501 20:17:28.340133 27525 raft_consensus.cc:3032] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:17:28.340230 27525 raft_consensus.cc:729] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:17:28.340540 27525 consensus_queue.cc:260] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:28.340999 27525 raft_consensus.cc:3037] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:17:28.342823 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d354124664bc0870f251d8a0831c5" candidate_uuid: "e677127814314770b3370764d0ff9d06" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:17:28.343065 27266 raft_consensus.cc:3037] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:17:28.353402 27525 raft_consensus.cc:2454] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 2.
I20240501 20:17:28.353992 27330 leader_election.cc:302] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e677127814314770b3370764d0ff9d06; no voters:
I20240501 20:17:28.354385 12973 raft_consensus.cc:2781] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:17:28.362829 27266 raft_consensus.cc:2454] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e677127814314770b3370764d0ff9d06 in term 2.
I20240501 20:17:28.390970 13057 tablet_replica.cc:330] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: stopping tablet replica
I20240501 20:17:28.391464 13057 raft_consensus.cc:2229] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:17:28.391886 13057 raft_consensus.cc:2258] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:17:28.393484 13057 ts_tablet_manager.cc:1875] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:17:28.395483 12973 raft_consensus.cc:686] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 2 LEADER]: Becoming Leader. State: Replica: e677127814314770b3370764d0ff9d06, State: Running, Role: LEADER
I20240501 20:17:28.395983 12973 consensus_queue.cc:237] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:28.317492 13053 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:28.403594 26988 catalog_manager.cc:5484] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 reported cstate change: term changed from 1 to 2, leader changed from 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) to e677127814314770b3370764d0ff9d06 (127.14.106.195). New cstate: current_term: 2 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:28.425040 27265 consensus_queue.cc:237] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
W20240501 20:17:28.431449 27202 consensus_peers.cc:485] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5b9f1a087114c82b010bd10e4f676c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:28.434374 29465 raft_consensus.cc:1261] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Refusing update from remote peer d8b307156e1346bd9eecf655f49924c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:17:28.435468 13055 consensus_queue.cc:1035] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:28.440750 29466 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:28.444475 13053 raft_consensus.cc:2932] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:28.447142 26988 catalog_manager.cc:5484] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 8 to 9, VOTER 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "d8b307156e1346bd9eecf655f49924c7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } }
I20240501 20:17:28.454352 26980 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e5b9f1a087114c82b010bd10e4f676c1 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:28.474381 13057 ts_tablet_manager.cc:1888] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 20:17:28.474664 13057 log.cc:1196] T e5b9f1a087114c82b010bd10e4f676c1 P 4d603ea5d17441b3bac562bf08ec83fe: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/e5b9f1a087114c82b010bd10e4f676c1
I20240501 20:17:28.475716 26980 catalog_manager.cc:4830] TS 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): tablet e5b9f1a087114c82b010bd10e4f676c1 (table rebalance_test_table_6 [id=fc800bf02dcd4a3d8f1c164f02b5fbe5]) successfully deleted
I20240501 20:17:28.527683 12968 ts_tablet_manager.cc:902] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: Initiating tablet copy from peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293)
I20240501 20:17:28.532212 12968 tablet_copy_client.cc:308] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46293
I20240501 20:17:28.533075 27154 tablet_copy_service.cc:140] P 4d603ea5d17441b3bac562bf08ec83fe: Received BeginTabletCopySession request for tablet 8bf74c24f28546bb90e10002270e3814 from peer 524f5e87a3ce40378bde76e667a4ece7 ({username='slave'} at 127.14.106.196:52025)
I20240501 20:17:28.533248 27154 tablet_copy_service.cc:161] P 4d603ea5d17441b3bac562bf08ec83fe: Beginning new tablet copy session on tablet 8bf74c24f28546bb90e10002270e3814 from peer 524f5e87a3ce40378bde76e667a4ece7 at {username='slave'} at 127.14.106.196:52025: session id = 524f5e87a3ce40378bde76e667a4ece7-8bf74c24f28546bb90e10002270e3814
I20240501 20:17:28.537156 27154 tablet_copy_source_session.cc:205] T 8bf74c24f28546bb90e10002270e3814 P 4d603ea5d17441b3bac562bf08ec83fe: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:28.541093 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf74c24f28546bb90e10002270e3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:28.559777 12968 tablet_copy_client.cc:783] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:28.586164 12968 tablet_copy_client.cc:647] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:28.588092 12968 tablet_copy_client.cc:523] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:28.593626 12968 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap starting.
I20240501 20:17:28.672544 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:28.683775 27504 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:17:28.700675 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:28.708410 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:28.730479 12968 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:28.745921 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:28.746496 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:28.752195 12968 tablet_bootstrap.cc:492] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: Bootstrap complete.
I20240501 20:17:28.752470 12968 ts_tablet_manager.cc:1367] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: Time spent bootstrapping tablet: real 0.159s user 0.023s sys 0.004s
I20240501 20:17:28.753158 12968 raft_consensus.cc:348] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:28.753378 12968 raft_consensus.cc:729] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Initialized, Role: FOLLOWER
I20240501 20:17:28.760982 12968 consensus_queue.cc:260] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:28.763373 12968 ts_tablet_manager.cc:1398] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:17:28.764041 27154 tablet_copy_service.cc:342] P 4d603ea5d17441b3bac562bf08ec83fe: Request end of tablet copy session 524f5e87a3ce40378bde76e667a4ece7-8bf74c24f28546bb90e10002270e3814 received from {username='slave'} at 127.14.106.196:52025
I20240501 20:17:28.764184 27154 tablet_copy_service.cc:433] P 4d603ea5d17441b3bac562bf08ec83fe: ending tablet copy session 524f5e87a3ce40378bde76e667a4ece7-8bf74c24f28546bb90e10002270e3814 on tablet 8bf74c24f28546bb90e10002270e3814 with peer 524f5e87a3ce40378bde76e667a4ece7
I20240501 20:17:28.801081 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:28.812638 27525 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:17:28.813285 13063 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:28.832748 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:28.837179 27265 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:17:28.837785 13063 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:28.841408 29050 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:28.858458 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:28.898034 27572 maintenance_manager.cc:419] P 524f5e87a3ce40378bde76e667a4ece7: Scheduling UndoDeltaBlockGCOp(8bf74c24f28546bb90e10002270e3814): 4834 bytes on disk
I20240501 20:17:28.898761 27463 maintenance_manager.cc:643] P 524f5e87a3ce40378bde76e667a4ece7: UndoDeltaBlockGCOp(8bf74c24f28546bb90e10002270e3814) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 20:17:28.901643 27504 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:17:28.915205 13133 ts_tablet_manager.cc:902] T e5b9f1a087114c82b010bd10e4f676c1 P 565cade0058540a4aa18d7c7a25affbe: Initiating tablet copy from peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
I20240501 20:17:28.918383 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:28.929348 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
F20240501 20:17:28.916028 13133 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0c836bb145a54160966c475e1490da38.queue_time_us, tablet-download-pool-0c836bb145a54160966c475e1490da38.queue_time_us) (tablet-download-pool-0c836bb145a54160966c475e1490da38.run_cpu_time_us, tablet-download-pool-0c836bb145a54160966c475e1490da38.run_cpu_time_us) (tablet-download-pool-0c836bb145a54160966c475e1490da38.run_wall_time_us, tablet-download-pool-0c836bb145a54160966c475e1490da38.run_wall_time_us) (tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.queue_time_us, tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.queue_time_us) (tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.run_cpu_time_us, tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.run_cpu_time_us) (tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.run_wall_time_us, tablet-download-pool-17e69dbd245e4b8db397aaffc9145c9f.run_wall_time_us) (tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.queue_time_us, tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.queue_time_us) (tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.run_cpu_time_us, tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.run_cpu_time_us) (tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.run_wall_time_us, tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.run_wall_time_us) (tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.queue_time_us, tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.queue_time_us) (tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.run_cpu_time_us, tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.run_cpu_time_us) (tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.run_wall_time_us, tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.run_wall_time_us) (tablet-download-pool-553d443dfb344b2690c260d207901ff1.queue_time_us, tablet-download-pool-553d443dfb344b2690c260d207901ff1.queue_time_us) (tablet-download-pool-553d443dfb344b2690c260d207901ff1.run_cpu_time_us, tablet-download-pool-553d443dfb344b2690c260d207901ff1.run_cpu_time_us) (tablet-download-pool-553d443dfb344b2690c260d207901ff1.run_wall_time_us, tablet-download-pool-553d443dfb344b2690c260d207901ff1.run_wall_time_us) (tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.queue_time_us, tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.queue_time_us) (tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.run_cpu_time_us, tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.run_cpu_time_us) (tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.run_wall_time_us, tablet-download-pool-6c836d9afa0f439b978bc2fe5b5e499d.run_wall_time_us) (tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.queue_time_us, tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.queue_time_us) (tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.run_cpu_time_us, tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.run_cpu_time_us) (tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.run_wall_time_us, tablet-download-pool-74c414c2eed045bda9d85fe5c2fea654.run_wall_time_us) (tablet-download-pool-af693dc19f844b12973b000093021619.queue_time_us, tablet-download-pool-af693dc19f844b12973b000093021619.queue_time_us) (tablet-download-pool-af693dc19f844b12973b000093021619.run_cpu_time_us, tablet-download-pool-af693dc19f844b12973b000093021619.run_cpu_time_us) (tablet-download-pool-af693dc19f844b12973b000093021619.run_wall_time_us, tablet-download-pool-af693dc19f844b12973b000093021619.run_wall_time_us) (tablet-download-pool-b5427fbc8a244a69974fd8d0f5f3d08d.queue_time_us, tablet-download-pool-b5427fbc8a244a69974fd8d0f5f3d08d.queue_time_us) (tablet-download-pool-b5427fbc8a244a69974fd8d0f5f3d08d.run_cpu_time_us, tablet-download-pool-b5427fbc8a244a69974fd8d0f5f3d08d.run_cpu_time_us) (tablet-download-pool-b5427fbc8a244a69974fd8d0f5f3d08d.run_wall_time_us, tablet-download-pool-b5427fbc8a244a69974fd8d0f5f3d08d.run_wall_time_us) (tablet-download-pool-b5a20d47052f404986315ddb05284ad9.queue_time_us, tablet-download-pool-b5a20d47052f404986315ddb05284ad9.queue_time_us) (tablet-download-pool-b5a20d47052f404986315ddb05284ad9.run_cpu_time_us, tablet-download-pool-b5a20d47052f404986315ddb05284ad9.run_cpu_time_us) (tablet-download-pool-b5a20d47052f404986315ddb05284ad9.run_wall_time_us, tablet-download-pool-b5a20d47052f404986315ddb05284ad9.run_wall_time_us) (tablet-download-pool-c2432c8e8e8847fa977055be919e79d6.queue_time_us, tablet-download-pool-c2432c8e8e8847fa977055be919e79d6.queue_time_us) (tablet-download-pool-c2432c8e8e8847fa977055be919e79d6.run_cpu_time_us, tablet-download-pool-c2432c8e8e8847fa977055be919e79d6.run_cpu_time_us) (tablet-download-pool-c2432c8e8e8847fa977055be919e79d6.run_wall_time_us, tablet-download-pool-c2432c8e8e8847fa977055be919e79d6.run_wall_time_us) (tablet-download-pool-d2160f17e62749dbb41405a895516b06.queue_time_us, tablet-download-pool-d2160f17e62749dbb41405a895516b06.queue_time_us) (tablet-download-pool-d2160f17e62749dbb41405a895516b06.run_cpu_time_us, tablet-download-pool-d2160f17e62749dbb41405a895516b06.run_cpu_time_us) (tablet-download-pool-d2160f17e62749dbb41405a895516b06.run_wall_time_us, tablet-download-pool-d2160f17e62749dbb41405a895516b06.run_wall_time_us) (tablet-download-pool-e5b9f1a087114c82b010bd10e4f676c1.queue_time_us, tablet-download-pool-e5b9f1a087114c82b010bd10e4f676c1.queue_time_us) (tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.queue_time_us, tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.queue_time_us) (tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.run_cpu_time_us, tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.run_cpu_time_us) (tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.run_wall_time_us, tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.run_wall_time_us) (tablet-download-pool-f034a679870b44caa238f04a8b49d302.queue_time_us, tablet-download-pool-f034a679870b44caa238f04a8b49d302.queue_time_us) (tablet-download-pool-f034a679870b44caa238f04a8b49d302.run_cpu_time_us, tablet-download-pool-f034a679870b44caa238f04a8b49d302.run_cpu_time_us) (tablet-download-pool-f034a679870b44caa238f04a8b49d302.run_wall_time_us, tablet-download-pool-f034a679870b44caa238f04a8b49d302.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 20:17:28.960347 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f241e378dcd google::LogMessage::Fail() at ??:0
@ 0x7f241e37cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f241e3787cc google::LogMessage::Flush() at ??:0
@ 0x7f241e379f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f24210b9742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f2421085f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 20:17:29.104372 27524 raft_consensus.cc:1203] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
@ 0x7f242107fcd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f24304d28de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f24304eadaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f2430615480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f243064da8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f2430630981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f243063070e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f2421089b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f242108d19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f242108d019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f242106801e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f24243ed6db start_thread at ??:0
@ 0x7f241d22b71f clone at ??:0
W20240501 20:17:29.224329 26979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 20:17:29.232904 26979 catalog_manager.cc:4573] TS d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c619a2235c9248eeb6b2e1d78583d1ba: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:29.255528 29517 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8oH_XT/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 20:11:45.395946 29524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:45.396442 29524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:45.414485 29524 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:48.751623 29524 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:11:50.820106 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.820390 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.820526 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.820693 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.820843 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.821096 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.821218 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.821424 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.821522 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.821656 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.821786 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.821947 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.822033 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.822129 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
I20240501 20:11:51.103153 29524 rebalancer_tool.cc:1074] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:51.441609 29524 rebalancer_tool.cc:1074] tablet b5427fbc8a244a69974fd8d0f5f3d08d: 'e677127814314770b3370764d0ff9d06' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:51.992283 29524 rebalancer_tool.cc:1074] tablet 8bf74c24f28546bb90e10002270e3814: 'd8b307156e1346bd9eecf655f49924c7' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:52.658437 29524 rebalancer_tool.cc:1074] tablet b5a20d47052f404986315ddb05284ad9: '4d603ea5d17441b3bac562bf08ec83fe' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:53.042416 29524 rebalancer_tool.cc:1084] tablet f034a679870b44caa238f04a8b49d302: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 20:11:53.042609 29524 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:54.614157 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:54.614331 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:54.614579 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:54.621834 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:54.623034 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:54.623253 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:54.623690 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:54.623903 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:54.623996 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:54.624105 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:54.624202 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:54.631947 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:54.632148 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:54.632277 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:54.632375 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:54.632474 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
I20240501 20:11:54.714022 29524 rebalancer_tool.cc:1074] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:11:55.264015 29524 rebalancer_tool.cc:1084] tablet 00e3777032eb4d9c80f47855e5f9f87a: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 20:11:55.264176 29524 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:56.664985 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:56.666252 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:56.666463 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:56.666718 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:56.666875 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:56.667058 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:56.667254 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:56.667434 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:56.667551 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:56.667659 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:56.669979 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
I20240501 20:11:56.950189 29524 rebalancer_tool.cc:1074] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:11:57.598387 29524 rebalancer_tool.cc:1074] tablet c2432c8e8e8847fa977055be919e79d6: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:58.213718 29524 rebalancer_tool.cc:1074] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:58.805459 29524 rebalancer_tool.cc:1084] tablet ebcdab26b4ed45f596282d4805d1bb1f: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 20:11:58.805614 29524 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:12:00.026774 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:12:00.029799 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:00.030741 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:12:00.031919 29524 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
I20240501 20:12:00.177366 29524 rebalancer_tool.cc:1074] tablet 00e3777032eb4d9c80f47855e5f9f87a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:00.690402 29524 rebalancer_tool.cc:1074] tablet 17e69dbd245e4b8db397aaffc9145c9f: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:12:00.872151 29524 rebalancer_tool.cc:1074] tablet 4dc5794a67374cc388da2edc47cbb113: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:00.987262 29524 rebalancer_tool.cc:1074] tablet 74c414c2eed045bda9d85fe5c2fea654: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:08.057369 29524 rebalancer_tool.cc:1123] tablet 00e3777032eb4d9c80f47855e5f9f87a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move is abandoned: Not found: Server with UUID 4d603ea5d17441b3bac562bf08ec83fe not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:17.284897 29524 rebalancer_tool.cc:1136] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:23.433926 29524 rebalancer_tool.cc:1123] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move is abandoned: Not found: Server with UUID 524f5e87a3ce40378bde76e667a4ece7 not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:23.515810 29524 rebalancer_tool.cc:1136] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:25.566810 29524 rebalancer_tool.cc:1123] tablet 8bf74c24f28546bb90e10002270e3814: 'd8b307156e1346bd9eecf655f49924c7' -> 'e957d29835544ac587917cc067fb7423' move is abandoned: Not found: Server with UUID d8b307156e1346bd9eecf655f49924c7 not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:12:25.686475 29524 rebalancer_tool.cc:1074] tablet af693dc19f844b12973b000093021619: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:25.817979 29524 rebalancer_tool.cc:1074] tablet c0e88937e1ec49bfbaffe535e445e40a: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:27.295101 29524 rebalancer_tool.cc:1136] tablet af693dc19f844b12973b000093021619: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:27.482787 29524 rebalancer_tool.cc:1136] tablet 4dc5794a67374cc388da2edc47cbb113: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:27.639416 29524 rebalancer_tool.cc:1136] tablet 74c414c2eed045bda9d85fe5c2fea654: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:27.696918 29524 rebalancer_tool.cc:1136] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:27.779172 29524 rebalancer_tool.cc:1136] tablet c2432c8e8e8847fa977055be919e79d6: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:27.830586 29524 rebalancer_tool.cc:1136] tablet b5427fbc8a244a69974fd8d0f5f3d08d: 'e677127814314770b3370764d0ff9d06' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:27.884479 29524 rebalancer_tool.cc:1136] tablet b5a20d47052f404986315ddb05284ad9: '4d603ea5d17441b3bac562bf08ec83fe' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:27.983384 29524 rebalancer_tool.cc:1136] tablet 17e69dbd245e4b8db397aaffc9145c9f: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:28.169528 29524 rebalancer_tool.cc:1074] tablet f02a619bc1584d53ac902c8d8baff976: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:28.304054 29524 rebalancer_tool.cc:1074] tablet 26cb2329d176441ca9127187db2d157a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:28.384792 29524 rebalancer_tool.cc:1074] tablet e5b9f1a087114c82b010bd10e4f676c1: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:28.461869 29524 rebalancer_tool.cc:1136] tablet e5b9f1a087114c82b010bd10e4f676c1: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:28.642900 29524 rebalancer_tool.cc:1136] tablet 26cb2329d176441ca9127187db2d157a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:28.827848 29524 rebalancer_tool.cc:1136] tablet f02a619bc1584d53ac902c8d8baff976: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:28.948386 29524 rebalancer_tool.cc:1136] tablet c0e88937e1ec49bfbaffe535e445e40a: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:29.164054 29524 rebalancer_tool.cc:1074] tablet 0c836bb145a54160966c475e1490da38: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:29.230706 29524 rebalancer_tool.cc:1074] tablet e7f03e02d13f44d188e5eb2774771a0d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:29.403461 29524 rebalancer_tool.cc:1074] tablet f034a679870b44caa238f04a8b49d302: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:29.438594 29524 rebalancer_tool.cc:1074] tablet 57b4bd50bc8f4d50957bd0280224c420: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:32.416584 29524 rebalancer_tool.cc:1136] tablet e7f03e02d13f44d188e5eb2774771a0d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:32.477995 29524 rebalancer_tool.cc:1136] tablet 0c836bb145a54160966c475e1490da38: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:32.563477 29524 rebalancer_tool.cc:1074] tablet 5272c34458af43bebb5ba5baefac7137: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:32.651044 29524 rebalancer_tool.cc:1074] tablet c619a2235c9248eeb6b2e1d78583d1ba: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:32.893086 29524 rebalancer_tool.cc:1136] tablet c619a2235c9248eeb6b2e1d78583d1ba: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:33.031469 29524 rebalancer_tool.cc:1136] tablet 5272c34458af43bebb5ba5baefac7137: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:36.154306 29524 rebalancer_tool.cc:1074] tablet 309e90bbd9f140a8beca9b3da3f75db4: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:36.274732 29524 rebalancer_tool.cc:1074] tablet 553d443dfb344b2690c260d207901ff1: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:36.517175 29524 rebalancer_tool.cc:1136] tablet 553d443dfb344b2690c260d207901ff1: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:36.602906 29524 rebalancer_tool.cc:1136] tablet 309e90bbd9f140a8beca9b3da3f75db4: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:37.854020 29524 rebalancer_tool.cc:1136] tablet f034a679870b44caa238f04a8b49d302: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
W20240501 20:16:42.138958 6890 meta_cache.cc:1261] Time spent looking up entry by key: real 0.062s user 0.001s sys 0.000s
W20240501 20:16:42.141999 6891 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.000s sys 0.000s
I20240501 20:17:27.145684 29524 rebalancer_tool.cc:1136] tablet 57b4bd50bc8f4d50957bd0280224c420: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
Illegal state: tablet server d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): unacceptable health status UNAVAILABLE
I20240501 20:17:29.461871 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:29.467694 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:29.502069 27504 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:17:29.541023 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:29.630168 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:29.631199 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:29.704823 29272 connection.cc:535] client connection to 127.14.106.197:40723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:29.705103 29272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:29.711016 27202 connection.cc:535] client connection to 127.14.106.197:40723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:29.711169 27202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 8 similar messages]
W20240501 20:17:29.725462 27462 connection.cc:535] client connection to 127.14.106.197:40723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:29.725469 27071 connection.cc:535] client connection to 127.14.106.197:40723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:29.725585 27462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 54 similar messages]
W20240501 20:17:29.725601 27071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 30 similar messages]
W20240501 20:17:29.735100 27332 connection.cc:535] client connection to 127.14.106.197:40723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:29.735253 27332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 10 similar messages]
W20240501 20:17:29.773517 27202 consensus_peers.cc:462] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
W20240501 20:17:29.773687 27202 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.774999 27071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 49376.2us
W20240501 20:17:29.781764 29272 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 76720.6us
W20240501 20:17:29.789075 27462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.197:40723, user_credentials={real_user=slave}} blocked reactor thread for 63458.7us
W20240501 20:17:29.789356 27332 consensus_peers.cc:485] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.789490 27332 consensus_peers.cc:485] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.789577 27332 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.792150 27071 consensus_peers.cc:485] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.792291 27071 consensus_peers.cc:485] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.792382 27071 consensus_peers.cc:485] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.798028 29272 consensus_peers.cc:485] T f034a679870b44caa238f04a8b49d302 P 212de34e3aa24fa1be12c7421df985fc -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.798352 27071 consensus_peers.cc:485] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.801719 27462 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.801894 27462 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.802001 27462 consensus_peers.cc:485] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.802110 27462 consensus_peers.cc:485] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:29.821666 27202 consensus_peers.cc:462] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:29.839653 27266 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26cb2329d176441ca9127187db2d157a"
dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59302
I20240501 20:17:29.839928 27266 raft_consensus.cc:595] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:17:29.926882 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:29.969877 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:30.009580 27246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:17:30.023000 27504 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:17:30.049706 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:30.051268 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:30.102542 13269 raft_consensus.cc:979] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: : Instructing follower 524f5e87a3ce40378bde76e667a4ece7 to start an election
I20240501 20:17:30.102754 13053 raft_consensus.cc:1067] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Signalling peer 524f5e87a3ce40378bde76e667a4ece7 to start an election
I20240501 20:17:30.103658 27526 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26cb2329d176441ca9127187db2d157a"
dest_uuid: "524f5e87a3ce40378bde76e667a4ece7"
from {username='slave'} at 127.14.106.194:53599
I20240501 20:17:30.103919 27526 raft_consensus.cc:482] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:17:30.104038 27526 raft_consensus.cc:3037] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:17:30.107661 27526 raft_consensus.cc:504] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:30.109072 27526 leader_election.cc:288] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 2 election: Requested vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), e957d29835544ac587917cc067fb7423 (127.14.106.200:44949)
I20240501 20:17:30.109467 29466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cb2329d176441ca9127187db2d157a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "e957d29835544ac587917cc067fb7423"
I20240501 20:17:30.109822 29466 raft_consensus.cc:3037] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:17:30.112531 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cb2329d176441ca9127187db2d157a" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "d8b307156e1346bd9eecf655f49924c7"
I20240501 20:17:30.112746 27266 raft_consensus.cc:3032] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:17:30.112825 27266 raft_consensus.cc:729] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8b307156e1346bd9eecf655f49924c7, State: Running, Role: LEADER
I20240501 20:17:30.113088 29466 raft_consensus.cc:2454] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 2.
I20240501 20:17:30.113130 27266 consensus_queue.cc:260] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:30.113564 27266 raft_consensus.cc:3037] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:17:30.113680 27461 leader_election.cc:302] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e957d29835544ac587917cc067fb7423; no voters:
I20240501 20:17:30.113973 13245 raft_consensus.cc:2781] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:17:30.114094 13245 raft_consensus.cc:686] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 2 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:17:30.114336 13245 consensus_queue.cc:237] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:30.117002 27266 raft_consensus.cc:2454] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 2.
I20240501 20:17:30.121083 26989 catalog_manager.cc:5484] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 1 to 2, leader changed from d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 2 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:30.210806 29519 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8oH_XT/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 20:11:45.097972 29520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:45.098410 29520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:45.111840 29520 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:48.677558 29520 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:11:50.775635 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.776154 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.776353 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.776455 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.776547 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.776638 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.776721 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.776888 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.776984 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.777076 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.777168 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.777274 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.777397 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.777514 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
I20240501 20:11:51.102541 29520 rebalancer_tool.cc:1084] tablet 309e90bbd9f140a8beca9b3da3f75db4: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 20:11:51.102766 29520 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:53.126591 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.128371 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:53.128460 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:53.129276 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:53.129411 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.129544 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:53.130142 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.130276 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.130386 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:53.130522 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.130718 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.130873 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:53.131040 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:53.132337 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.132439 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:53.132552 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.132675 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:53.444303 29520 rebalancer_tool.cc:1074] tablet 74c414c2eed045bda9d85fe5c2fea654: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:53.840773 29520 rebalancer_tool.cc:1074] tablet c0e88937e1ec49bfbaffe535e445e40a: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:54.555449 29520 rebalancer_tool.cc:1074] tablet c619a2235c9248eeb6b2e1d78583d1ba: 'd8b307156e1346bd9eecf655f49924c7' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:54.950942 29520 rebalancer_tool.cc:1074] tablet af693dc19f844b12973b000093021619: 'e677127814314770b3370764d0ff9d06' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:55.377498 29520 rebalancer_tool.cc:1074] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:55.426311 29520 rebalancer_tool.cc:1074] tablet 4dc5794a67374cc388da2edc47cbb113: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:11:55.892391 29520 rebalancer_tool.cc:1074] tablet 57b4bd50bc8f4d50957bd0280224c420: '524f5e87a3ce40378bde76e667a4ece7' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:56.685181 29520 rebalancer_tool.cc:1074] tablet 3f8902b7b0b649edb10f362713ad36fe: 'e677127814314770b3370764d0ff9d06' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:56.721802 29520 rebalancer_tool.cc:1084] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move ignored: Illegal state: tablet 7bb13c1dce8640b28063daa0132d2896: replica d56cbcb06b9e418cb617de6a34b946b0 is already present
I20240501 20:11:56.721958 29520 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:58.504951 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:58.524931 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:58.530799 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:58.531029 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:58.531164 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:58.534739 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:58.534972 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:58.535656 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:58.535941 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:58.541800 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
I20240501 20:11:58.665514 29520 rebalancer_tool.cc:1074] tablet e7f03e02d13f44d188e5eb2774771a0d: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:11:59.093191 29520 rebalancer_tool.cc:1084] tablet d2160f17e62749dbb41405a895516b06: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 20:11:59.093370 29520 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:12:01.095683 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:12:01.095855 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:12:01.096043 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:01.096148 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:01.096228 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:01.096369 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:12:01.096505 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:01.096729 29520 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
I20240501 20:12:01.229108 29520 rebalancer_tool.cc:1074] tablet 00e3777032eb4d9c80f47855e5f9f87a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:01.635421 29520 rebalancer_tool.cc:1074] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:01.843300 29520 rebalancer_tool.cc:1074] tablet e5b9f1a087114c82b010bd10e4f676c1: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:15.479584 29520 rebalancer_tool.cc:1136] tablet 00e3777032eb4d9c80f47855e5f9f87a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:21.782541 29520 rebalancer_tool.cc:1136] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:23.736397 29520 rebalancer_tool.cc:1136] tablet af693dc19f844b12973b000093021619: 'e677127814314770b3370764d0ff9d06' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:23.908789 29520 rebalancer_tool.cc:1074] tablet 26cb2329d176441ca9127187db2d157a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:24.064200 29520 rebalancer_tool.cc:1074] tablet 553d443dfb344b2690c260d207901ff1: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:24.160264 29520 rebalancer_tool.cc:1136] tablet 553d443dfb344b2690c260d207901ff1: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:24.234313 29520 rebalancer_tool.cc:1136] tablet 26cb2329d176441ca9127187db2d157a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:24.293087 29520 rebalancer_tool.cc:1136] tablet e5b9f1a087114c82b010bd10e4f676c1: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:24.332355 29520 rebalancer_tool.cc:1136] tablet e7f03e02d13f44d188e5eb2774771a0d: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:24.405925 29520 rebalancer_tool.cc:1136] tablet 57b4bd50bc8f4d50957bd0280224c420: '524f5e87a3ce40378bde76e667a4ece7' -> 'e957d29835544ac587917cc067fb7423' move completed: OK
I20240501 20:12:24.621989 29520 rebalancer_tool.cc:1136] tablet 3f8902b7b0b649edb10f362713ad36fe: 'e677127814314770b3370764d0ff9d06' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:24.675618 29520 rebalancer_tool.cc:1136] tablet c0e88937e1ec49bfbaffe535e445e40a: '4d603ea5d17441b3bac562bf08ec83fe' -> 'e957d29835544ac587917cc067fb7423' move completed: OK
I20240501 20:12:24.733754 29520 rebalancer_tool.cc:1136] tablet 4dc5794a67374cc388da2edc47cbb113: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:24.857980 29520 rebalancer_tool.cc:1136] tablet 74c414c2eed045bda9d85fe5c2fea654: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:24.882246 29520 rebalancer_tool.cc:1136] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:24.948601 29520 rebalancer_tool.cc:1136] tablet c619a2235c9248eeb6b2e1d78583d1ba: 'd8b307156e1346bd9eecf655f49924c7' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:25.254892 29520 rebalancer_tool.cc:1074] tablet b5427fbc8a244a69974fd8d0f5f3d08d: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:25.371455 29520 rebalancer_tool.cc:1074] tablet f034a679870b44caa238f04a8b49d302: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:25.440532 29520 rebalancer_tool.cc:1074] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:25.526895 29520 rebalancer_tool.cc:1074] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:25.723212 29520 rebalancer_tool.cc:1074] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:25.898023 29520 rebalancer_tool.cc:1136] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:26.060271 29520 rebalancer_tool.cc:1136] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:26.140133 29520 rebalancer_tool.cc:1136] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:29.002247 29520 rebalancer_tool.cc:1136] tablet b5427fbc8a244a69974fd8d0f5f3d08d: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:29.159307 29520 rebalancer_tool.cc:1074] tablet 8bf74c24f28546bb90e10002270e3814: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:29.226460 29520 rebalancer_tool.cc:1074] tablet b5a20d47052f404986315ddb05284ad9: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:29.290969 29520 rebalancer_tool.cc:1136] tablet b5a20d47052f404986315ddb05284ad9: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:33.556663 29520 rebalancer_tool.cc:1136] tablet f034a679870b44caa238f04a8b49d302: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:17:28.193519 29520 rebalancer_tool.cc:1136] tablet 8bf74c24f28546bb90e10002270e3814: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
Illegal state: tablet server d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): unacceptable health status UNAVAILABLE
I20240501 20:17:30.240561 29316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:17:30.255173 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:30.256584 13260 raft_consensus.cc:482] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 565cade0058540a4aa18d7c7a25affbe)
I20240501 20:17:30.256778 13260 raft_consensus.cc:504] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:30.257485 13260 leader_election.cc:288] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723), 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
I20240501 20:17:30.258905 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:30.259456 29334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "4d603ea5d17441b3bac562bf08ec83fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "212de34e3aa24fa1be12c7421df985fc" is_pre_election: true
W20240501 20:17:30.273255 27071 leader_election.cc:334] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:30.273412 27071 leader_election.cc:302] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe; no voters: 212de34e3aa24fa1be12c7421df985fc, 565cade0058540a4aa18d7c7a25affbe
I20240501 20:17:30.274053 12896 raft_consensus.cc:2726] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:17:30.294888 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:17:30.321084 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:30.335896 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:30.342667 12742 raft_consensus.cc:482] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 565cade0058540a4aa18d7c7a25affbe)
I20240501 20:17:30.342886 12742 raft_consensus.cc:504] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:30.343817 12742 leader_election.cc:288] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
W20240501 20:17:30.351022 29272 leader_election.cc:334] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:30.353250 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "212de34e3aa24fa1be12c7421df985fc" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe" is_pre_election: true
I20240501 20:17:30.353516 27135 raft_consensus.cc:2454] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212de34e3aa24fa1be12c7421df985fc in term 2.
I20240501 20:17:30.354049 29271 leader_election.cc:302] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212de34e3aa24fa1be12c7421df985fc, 4d603ea5d17441b3bac562bf08ec83fe; no voters: 565cade0058540a4aa18d7c7a25affbe
I20240501 20:17:30.354485 12742 raft_consensus.cc:2781] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:17:30.354568 12742 raft_consensus.cc:482] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Starting leader election (detected failure of leader 565cade0058540a4aa18d7c7a25affbe)
I20240501 20:17:30.354661 12742 raft_consensus.cc:3037] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:17:30.359230 12742 raft_consensus.cc:504] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:30.360474 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d443dfb344b2690c260d207901ff1" candidate_uuid: "212de34e3aa24fa1be12c7421df985fc" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "4d603ea5d17441b3bac562bf08ec83fe"
I20240501 20:17:30.360699 27135 raft_consensus.cc:3037] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:17:30.364009 27135 raft_consensus.cc:2454] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212de34e3aa24fa1be12c7421df985fc in term 3.
I20240501 20:17:30.364499 29271 leader_election.cc:302] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212de34e3aa24fa1be12c7421df985fc, 4d603ea5d17441b3bac562bf08ec83fe; no voters:
I20240501 20:17:30.365068 13298 raft_consensus.cc:2781] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 3 FOLLOWER]: Leader election won for term 3
W20240501 20:17:30.367105 29272 leader_election.cc:334] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:30.367380 12742 leader_election.cc:288] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [CANDIDATE]: Term 3 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
I20240501 20:17:30.399000 13298 raft_consensus.cc:686] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [term 3 LEADER]: Becoming Leader. State: Replica: 212de34e3aa24fa1be12c7421df985fc, State: Running, Role: LEADER
I20240501 20:17:30.399320 13298 consensus_queue.cc:237] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:30.403025 26985 catalog_manager.cc:5484] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc reported cstate change: term changed from 2 to 3, leader changed from 565cade0058540a4aa18d7c7a25affbe (127.14.106.197) to 212de34e3aa24fa1be12c7421df985fc (127.14.106.199). New cstate: current_term: 3 leader_uuid: "212de34e3aa24fa1be12c7421df985fc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
W20240501 20:17:30.424712 27202 consensus_peers.cc:462] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:30.435899 27395 consensus_queue.cc:237] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } }
I20240501 20:17:30.438237 27266 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:17:30.438836 13224 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:30.441346 12973 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:30.441828 27266 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:30.446954 26985 catalog_manager.cc:5484] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 reported cstate change: config changed from index 7 to 9, VOTER 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196) evicted. New cstate: current_term: 2 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } }
I20240501 20:17:30.453401 27395 consensus_queue.cc:237] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:30.453562 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 944d354124664bc0870f251d8a0831c5 with delete_type TABLET_DATA_TOMBSTONED (TS 524f5e87a3ce40378bde76e667a4ece7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50208
W20240501 20:17:30.457991 27329 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 944d354124664bc0870f251d8a0831c5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:30.460641 13321 tablet_replica.cc:330] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: stopping tablet replica
I20240501 20:17:30.460897 13321 raft_consensus.cc:2229] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:17:30.461261 13321 raft_consensus.cc:2258] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:17:30.461421 27266 raft_consensus.cc:1261] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Refusing update from remote peer e677127814314770b3370764d0ff9d06: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:17:30.462028 12973 consensus_queue.cc:1035] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:17:30.462455 13321 ts_tablet_manager.cc:1875] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:17:30.464326 13320 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:30.465294 27266 raft_consensus.cc:2932] T 944d354124664bc0870f251d8a0831c5 P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:30.468780 26981 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 944d354124664bc0870f251d8a0831c5 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:30.471123 26985 catalog_manager.cc:5484] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 reported cstate change: config changed from index 9 to 10, VOTER 212de34e3aa24fa1be12c7421df985fc (127.14.106.199) added. New cstate: current_term: 2 leader_uuid: "e677127814314770b3370764d0ff9d06" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } }
I20240501 20:17:30.475126 13321 ts_tablet_manager.cc:1888] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 20:17:30.475273 13321 log.cc:1196] T 944d354124664bc0870f251d8a0831c5 P 524f5e87a3ce40378bde76e667a4ece7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/wals/944d354124664bc0870f251d8a0831c5
I20240501 20:17:30.476251 26979 catalog_manager.cc:4830] TS 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861): tablet 944d354124664bc0870f251d8a0831c5 (table rebalance_test_table_7 [id=ff19b427845646c9a01ebd69c335a06b]) successfully deleted
I20240501 20:17:30.584089 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:30.585180 29316 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:17:30.602160 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:30.610810 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:30.613529 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:17:30.619129 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:30.688359 29466 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:17:30.689162 12579 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:17:30.711513 27266 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:17:30.712134 12579 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:17:30.884104 29272 consensus_peers.cc:485] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:30.911654 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:30.914675 29316 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:17:30.922310 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:30.945017 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:17:30.947204 27202 consensus_peers.cc:462] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:30.969609 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:17:30.983841 27135 raft_consensus.cc:1261] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 3 FOLLOWER]: Refusing update from remote peer 212de34e3aa24fa1be12c7421df985fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 20:17:30.984654 13298 consensus_queue.cc:1035] T 553d443dfb344b2690c260d207901ff1 P 212de34e3aa24fa1be12c7421df985fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:31.010941 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:31.064021 13414 ts_tablet_manager.cc:902] T 944d354124664bc0870f251d8a0831c5 P 212de34e3aa24fa1be12c7421df985fc: Initiating tablet copy from peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
F20240501 20:17:31.064479 13414 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00e3777032eb4d9c80f47855e5f9f87a.queue_time_us, tablet-download-pool-00e3777032eb4d9c80f47855e5f9f87a.queue_time_us) (tablet-download-pool-00e3777032eb4d9c80f47855e5f9f87a.run_cpu_time_us, tablet-download-pool-00e3777032eb4d9c80f47855e5f9f87a.run_cpu_time_us) (tablet-download-pool-00e3777032eb4d9c80f47855e5f9f87a.run_wall_time_us, tablet-download-pool-00e3777032eb4d9c80f47855e5f9f87a.run_wall_time_us) (tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.queue_time_us, tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.queue_time_us) (tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.run_cpu_time_us, tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.run_cpu_time_us) (tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.run_wall_time_us, tablet-download-pool-3f8902b7b0b649edb10f362713ad36fe.run_wall_time_us) (tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.queue_time_us, tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.queue_time_us) (tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.run_cpu_time_us, tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.run_cpu_time_us) (tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.run_wall_time_us, tablet-download-pool-4dc5794a67374cc388da2edc47cbb113.run_wall_time_us) (tablet-download-pool-50bf9058852a4ea49733326057d5ff41.queue_time_us, tablet-download-pool-50bf9058852a4ea49733326057d5ff41.queue_time_us) (tablet-download-pool-50bf9058852a4ea49733326057d5ff41.run_cpu_time_us, tablet-download-pool-50bf9058852a4ea49733326057d5ff41.run_cpu_time_us) (tablet-download-pool-50bf9058852a4ea49733326057d5ff41.run_wall_time_us, tablet-download-pool-50bf9058852a4ea49733326057d5ff41.run_wall_time_us) (tablet-download-pool-5272c34458af43bebb5ba5baefac7137.queue_time_us, tablet-download-pool-5272c34458af43bebb5ba5baefac7137.queue_time_us) (tablet-download-pool-5272c34458af43bebb5ba5baefac7137.run_cpu_time_us, tablet-download-pool-5272c34458af43bebb5ba5baefac7137.run_cpu_time_us) (tablet-download-pool-5272c34458af43bebb5ba5baefac7137.run_wall_time_us, tablet-download-pool-5272c34458af43bebb5ba5baefac7137.run_wall_time_us) (tablet-download-pool-553d443dfb344b2690c260d207901ff1.queue_time_us, tablet-download-pool-553d443dfb344b2690c260d207901ff1.queue_time_us) (tablet-download-pool-553d443dfb344b2690c260d207901ff1.run_cpu_time_us, tablet-download-pool-553d443dfb344b2690c260d207901ff1.run_cpu_time_us) (tablet-download-pool-553d443dfb344b2690c260d207901ff1.run_wall_time_us, tablet-download-pool-553d443dfb344b2690c260d207901ff1.run_wall_time_us) (tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.queue_time_us, tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.queue_time_us) (tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.run_cpu_time_us, tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.run_cpu_time_us) (tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.run_wall_time_us, tablet-download-pool-57b4bd50bc8f4d50957bd0280224c420.run_wall_time_us) (tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.queue_time_us, tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.queue_time_us) (tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.run_cpu_time_us, tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.run_cpu_time_us) (tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.run_wall_time_us, tablet-download-pool-78fcaa70e916442ea9c67cc762c8db6d.run_wall_time_us) (tablet-download-pool-944d354124664bc0870f251d8a0831c5.queue_time_us, tablet-download-pool-944d354124664bc0870f251d8a0831c5.queue_time_us) (tablet-download-pool-b5a20d47052f404986315ddb05284ad9.queue_time_us, tablet-download-pool-b5a20d47052f404986315ddb05284ad9.queue_time_us) (tablet-download-pool-b5a20d47052f404986315ddb05284ad9.run_cpu_time_us, tablet-download-pool-b5a20d47052f404986315ddb05284ad9.run_cpu_time_us) (tablet-download-pool-b5a20d47052f404986315ddb05284ad9.run_wall_time_us, tablet-download-pool-b5a20d47052f404986315ddb05284ad9.run_wall_time_us) (tablet-download-pool-c0e88937e1ec49bfbaffe535e445e40a.queue_time_us, tablet-download-pool-c0e88937e1ec49bfbaffe535e445e40a.queue_time_us) (tablet-download-pool-c0e88937e1ec49bfbaffe535e445e40a.run_cpu_time_us, tablet-download-pool-c0e88937e1ec49bfbaffe535e445e40a.run_cpu_time_us) (tablet-download-pool-c0e88937e1ec49bfbaffe535e445e40a.run_wall_time_us, tablet-download-pool-c0e88937e1ec49bfbaffe535e445e40a.run_wall_time_us) (tablet-download-pool-c619a2235c9248eeb6b2e1d78583d1ba.queue_time_us, tablet-download-pool-c619a2235c9248eeb6b2e1d78583d1ba.queue_time_us) (tablet-download-pool-c619a2235c9248eeb6b2e1d78583d1ba.run_cpu_time_us, tablet-download-pool-c619a2235c9248eeb6b2e1d78583d1ba.run_cpu_time_us) (tablet-download-pool-c619a2235c9248eeb6b2e1d78583d1ba.run_wall_time_us, tablet-download-pool-c619a2235c9248eeb6b2e1d78583d1ba.run_wall_time_us) (tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.queue_time_us, tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.queue_time_us) (tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.run_cpu_time_us, tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.run_cpu_time_us) (tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.run_wall_time_us, tablet-download-pool-e7f03e02d13f44d188e5eb2774771a0d.run_wall_time_us) (tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.queue_time_us, tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.queue_time_us) (tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.run_cpu_time_us, tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.run_cpu_time_us) (tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.run_wall_time_us, tablet-download-pool-f02a619bc1584d53ac902c8d8baff976.run_wall_time_us) (tablet-download-pool-f034a679870b44caa238f04a8b49d302.queue_time_us, tablet-download-pool-f034a679870b44caa238f04a8b49d302.queue_time_us) (tablet-download-pool-f034a679870b44caa238f04a8b49d302.run_cpu_time_us, tablet-download-pool-f034a679870b44caa238f04a8b49d302.run_cpu_time_us) (tablet-download-pool-f034a679870b44caa238f04a8b49d302.run_wall_time_us, tablet-download-pool-f034a679870b44caa238f04a8b49d302.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f19a19d0dcd google::LogMessage::Fail() at ??:0
@ 0x7f19a19d4b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f19a19d07cc google::LogMessage::Flush() at ??:0
@ 0x7f19a19d1f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f19a4711742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f19a46ddf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f19a46d7cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f19b3b2a8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f19b3b42daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f19b3c6d480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f19b3ca5a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f19b3c88981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f19b3c8870e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f19a46e1b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f19a46e519c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f19a46e5019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f19a46c001e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f19a7a456db start_thread at ??:0
@ 0x7f19a088371f clone at ??:0
W20240501 20:17:31.338018 26979 catalog_manager.cc:4573] TS d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c619a2235c9248eeb6b2e1d78583d1ba: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:31.413710 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:17:31.429510 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:17:31.434666 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:31.435997 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:17:31.439950 27332 consensus_peers.cc:485] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:17:31.464035 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:31.500030 27202 consensus_peers.cc:462] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
W20240501 20:17:31.549613 27459 connection.cc:535] client connection to 127.14.106.199:32969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:31.549651 27329 connection.cc:535] client connection to 127.14.106.199:32969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:31.551657 27068 connection.cc:535] client connection to 127.14.106.199:32969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:31.553493 27068 consensus_peers.cc:485] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.553639 27068 consensus_peers.cc:485] T 78fcaa70e916442ea9c67cc762c8db6d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.553751 27068 consensus_peers.cc:485] T 50bf9058852a4ea49733326057d5ff41 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.553859 27068 consensus_peers.cc:485] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.554260 13214 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:17:31.550572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.106.199:32969 (local address 127.14.106.195:33567)
0501 20:17:31.550745 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0501 20:17:31.550763 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0501 20:17:31.550896 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:17:31.554078 (+ 3182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":71}
W20240501 20:17:31.554445 27329 consensus_peers.cc:462] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:31.554539 27329 consensus_peers.cc:485] T b5a20d47052f404986315ddb05284ad9 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.554648 27329 consensus_peers.cc:485] T c619a2235c9248eeb6b2e1d78583d1ba P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.559873 27459 consensus_peers.cc:485] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.560014 27459 consensus_peers.cc:485] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.560109 27459 consensus_peers.cc:485] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.576179 27329 consensus_peers.cc:485] T 57b4bd50bc8f4d50957bd0280224c420 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:31.594735 27068 consensus_peers.cc:485] T e7f03e02d13f44d188e5eb2774771a0d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:31.605134 27526 consensus_queue.cc:237] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:31.606794 29465 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:17:31.607334 13245 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:17:31.609755 13491 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:31.610455 29465 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
I20240501 20:17:31.615247 26985 catalog_manager.cc:5484] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: config changed from index 7 to 9, VOTER d8b307156e1346bd9eecf655f49924c7 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } }
W20240501 20:17:31.618325 27202 consensus_peers.cc:485] T 74c414c2eed045bda9d85fe5c2fea654 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:31.621537 27329 consensus_peers.cc:462] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
I20240501 20:17:31.642019 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet 26cb2329d176441ca9127187db2d157a with delete_type TABLET_DATA_TOMBSTONED (TS d8b307156e1346bd9eecf655f49924c7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59150
I20240501 20:17:31.643379 27526 consensus_queue.cc:237] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:31.645728 29465 raft_consensus.cc:1261] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Refusing update from remote peer 524f5e87a3ce40378bde76e667a4ece7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240501 20:17:31.646100 27462 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Couldn't send request to peer e677127814314770b3370764d0ff9d06. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26cb2329d176441ca9127187db2d157a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:31.646397 13491 consensus_queue.cc:1035] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:17:31.649355 13245 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:31.649917 29465 raft_consensus.cc:2932] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:31.651037 13497 tablet_replica.cc:330] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: stopping tablet replica
I20240501 20:17:31.651252 13497 raft_consensus.cc:2229] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:17:31.651444 13497 raft_consensus.cc:2258] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:17:31.660035 26985 catalog_manager.cc:5484] T 26cb2329d176441ca9127187db2d157a P e957d29835544ac587917cc067fb7423 reported cstate change: config changed from index 9 to 10, VOTER e677127814314770b3370764d0ff9d06 (127.14.106.195) added. New cstate: current_term: 2 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } }
I20240501 20:17:31.663236 26979 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 26cb2329d176441ca9127187db2d157a with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:17:31.663368 13497 ts_tablet_manager.cc:1875] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:17:31.686245 13497 ts_tablet_manager.cc:1888] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 20:17:31.686503 13497 log.cc:1196] T 26cb2329d176441ca9127187db2d157a P d8b307156e1346bd9eecf655f49924c7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/26cb2329d176441ca9127187db2d157a
I20240501 20:17:31.688719 26980 catalog_manager.cc:4830] TS d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): tablet 26cb2329d176441ca9127187db2d157a (table rebalance_test_table_0 [id=565e816257734920a19d1af631224041]) successfully deleted
W20240501 20:17:31.704149 27462 consensus_peers.cc:485] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:31.704378 27068 consensus_peers.cc:485] T 5272c34458af43bebb5ba5baefac7137 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:31.804352 29516 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8oH_XT/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 20:11:45.350582 29522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:45.350989 29522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:45.365131 29522 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:48.743499 29522 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:11:50.644731 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.644927 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.645081 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.645246 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.645326 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.645516 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.645610 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.647652 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.647791 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.647914 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.648003 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.648113 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.648218 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.648315 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.648411 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
I20240501 20:11:50.922896 29522 rebalancer_tool.cc:1084] tablet 309e90bbd9f140a8beca9b3da3f75db4: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:11:50.923151 29522 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:53.241521 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.241755 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.241994 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.242183 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.242311 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.242442 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.242563 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.261090 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:53.261243 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:53.263120 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:53.263263 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:53.263394 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.263504 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:53.263598 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:53.268733 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:53.268914 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:53.269073 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:53.269193 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:53.631306 29522 rebalancer_tool.cc:1084] tablet 74c414c2eed045bda9d85fe5c2fea654: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 20:11:53.631433 29522 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:55.426837 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:55.427806 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:55.428042 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:55.428190 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:55.428443 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:55.428622 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:55.428745 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:55.428838 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:55.428949 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:55.429056 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:55.429224 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:55.552229 29522 rebalancer_tool.cc:1074] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:11:56.068470 29522 rebalancer_tool.cc:1074] tablet 553d443dfb344b2690c260d207901ff1: 'e677127814314770b3370764d0ff9d06' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:56.266875 29522 rebalancer_tool.cc:1074] tablet c0e88937e1ec49bfbaffe535e445e40a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:11:56.809074 29522 rebalancer_tool.cc:1074] tablet e5b9f1a087114c82b010bd10e4f676c1: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:57.135671 29522 rebalancer_tool.cc:1074] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:57.581941 29522 rebalancer_tool.cc:1074] tablet 5272c34458af43bebb5ba5baefac7137: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:58.220191 29522 rebalancer_tool.cc:1074] tablet 0c836bb145a54160966c475e1490da38: 'e677127814314770b3370764d0ff9d06' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:58.555989 29522 rebalancer_tool.cc:1084] tablet ebcdab26b4ed45f596282d4805d1bb1f: '4d603ea5d17441b3bac562bf08ec83fe' -> '565cade0058540a4aa18d7c7a25affbe' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 20:11:58.556097 29522 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:12:00.111423 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:12:00.111601 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:00.111752 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:00.111904 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
I20240501 20:12:00.306457 29522 rebalancer_tool.cc:1074] tablet 00e3777032eb4d9c80f47855e5f9f87a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:00.637228 29522 rebalancer_tool.cc:1084] tablet 17e69dbd245e4b8db397aaffc9145c9f: 'e677127814314770b3370764d0ff9d06' -> 'd56cbcb06b9e418cb617de6a34b946b0' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:00.637414 29522 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:12:02.191218 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:02.191462 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:02.191723 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:12:02.191874 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:02.193208 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:12:02.193547 29522 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
I20240501 20:12:02.594235 29522 rebalancer_tool.cc:1074] tablet e7f03e02d13f44d188e5eb2774771a0d: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:02.693055 29522 rebalancer_tool.cc:1074] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:12.740114 29522 rebalancer_tool.cc:1136] tablet e7f03e02d13f44d188e5eb2774771a0d: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:12.822048 29522 rebalancer_tool.cc:1136] tablet e5b9f1a087114c82b010bd10e4f676c1: '524f5e87a3ce40378bde76e667a4ece7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:17.390935 29522 rebalancer_tool.cc:1136] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> 'e957d29835544ac587917cc067fb7423' move completed: OK
I20240501 20:12:19.732584 29522 rebalancer_tool.cc:1074] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:22.107147 29522 rebalancer_tool.cc:1136] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:22.188721 29522 rebalancer_tool.cc:1136] tablet 00e3777032eb4d9c80f47855e5f9f87a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:22.420688 29522 rebalancer_tool.cc:1136] tablet 0c836bb145a54160966c475e1490da38: 'e677127814314770b3370764d0ff9d06' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:22.485996 29522 rebalancer_tool.cc:1136] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:22.573974 29522 rebalancer_tool.cc:1136] tablet c0e88937e1ec49bfbaffe535e445e40a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:22.688756 29522 rebalancer_tool.cc:1136] tablet 553d443dfb344b2690c260d207901ff1: 'e677127814314770b3370764d0ff9d06' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:22.967336 29522 rebalancer_tool.cc:1136] tablet 5272c34458af43bebb5ba5baefac7137: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:23.103632 29522 rebalancer_tool.cc:1074] tablet 3f8902b7b0b649edb10f362713ad36fe: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:23.185503 29522 rebalancer_tool.cc:1074] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:23.326179 29522 rebalancer_tool.cc:1074] tablet f034a679870b44caa238f04a8b49d302: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:23.437050 29522 rebalancer_tool.cc:1074] tablet c2432c8e8e8847fa977055be919e79d6: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:23.481019 29522 rebalancer_tool.cc:1136] tablet c2432c8e8e8847fa977055be919e79d6: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:23.537689 29522 rebalancer_tool.cc:1136] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:23.724339 29522 rebalancer_tool.cc:1136] tablet 3f8902b7b0b649edb10f362713ad36fe: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:23.800802 29522 rebalancer_tool.cc:1136] tablet f034a679870b44caa238f04a8b49d302: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:23.893379 29522 rebalancer_tool.cc:1136] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:23.988693 29522 rebalancer_tool.cc:1074] tablet 26cb2329d176441ca9127187db2d157a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:24.053675 29522 rebalancer_tool.cc:1074] tablet 944d354124664bc0870f251d8a0831c5: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:24.210552 29522 rebalancer_tool.cc:1074] tablet 74c414c2eed045bda9d85fe5c2fea654: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:24.279067 29522 rebalancer_tool.cc:1074] tablet b5a20d47052f404986315ddb05284ad9: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:24.387298 29522 rebalancer_tool.cc:1074] tablet c619a2235c9248eeb6b2e1d78583d1ba: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:28.173949 29522 rebalancer_tool.cc:1136] tablet 74c414c2eed045bda9d85fe5c2fea654: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:30.770679 29522 rebalancer_tool.cc:1136] tablet 26cb2329d176441ca9127187db2d157a: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:30.841003 29522 rebalancer_tool.cc:1136] tablet c619a2235c9248eeb6b2e1d78583d1ba: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:31.057051 29522 rebalancer_tool.cc:1136] tablet b5a20d47052f404986315ddb05284ad9: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:17:30.763589 29522 rebalancer_tool.cc:1136] tablet 944d354124664bc0870f251d8a0831c5: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
Illegal state: tablet server 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): unacceptable health status UNAVAILABLE
W20240501 20:17:31.850600 27071 consensus_peers.cc:485] T c2432c8e8e8847fa977055be919e79d6 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:31.890370 27071 consensus_peers.cc:485] T 3f8902b7b0b649edb10f362713ad36fe P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:17:31.959375 27504 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:17:31.967429 27376 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:17:31.968935 27202 consensus_peers.cc:462] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:31.971702 27113 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:17:31.996675 27071 consensus_peers.cc:485] T b5427fbc8a244a69974fd8d0f5f3d08d P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:17:31.997040 27246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:17:31.997203 29447 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:17:32.032967 27329 consensus_peers.cc:462] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
W20240501 20:17:32.047432 27332 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:32.075748 27332 consensus_peers.cc:485] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:32.081084 27462 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:32.103669 27462 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:32.154750 27462 consensus_peers.cc:485] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:17:32.166647 13550 ts_tablet_manager.cc:902] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: Initiating tablet copy from peer 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
I20240501 20:17:32.168040 13550 tablet_copy_client.cc:308] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.196:43861
I20240501 20:17:32.168713 27545 tablet_copy_service.cc:140] P 524f5e87a3ce40378bde76e667a4ece7: Received BeginTabletCopySession request for tablet 26cb2329d176441ca9127187db2d157a from peer e677127814314770b3370764d0ff9d06 ({username='slave'} at 127.14.106.195:33343)
I20240501 20:17:32.168895 27545 tablet_copy_service.cc:161] P 524f5e87a3ce40378bde76e667a4ece7: Beginning new tablet copy session on tablet 26cb2329d176441ca9127187db2d157a from peer e677127814314770b3370764d0ff9d06 at {username='slave'} at 127.14.106.195:33343: session id = e677127814314770b3370764d0ff9d06-26cb2329d176441ca9127187db2d157a
I20240501 20:17:32.171684 27545 tablet_copy_source_session.cc:205] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:17:32.172951 13550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cb2329d176441ca9127187db2d157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:32.179520 13550 tablet_copy_client.cc:783] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: tablet copy: Starting download of 5 data blocks...
I20240501 20:17:32.186255 13550 tablet_copy_client.cc:647] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: tablet copy: Starting download of 1 WAL segments...
I20240501 20:17:32.188333 13550 tablet_copy_client.cc:523] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:17:32.192459 13550 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: Bootstrap starting.
I20240501 20:17:32.214138 13550 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:17:32.214577 13550 tablet_bootstrap.cc:492] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: Bootstrap complete.
I20240501 20:17:32.214866 13550 ts_tablet_manager.cc:1367] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.001s
I20240501 20:17:32.215786 13550 raft_consensus.cc:348] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:32.216030 13550 raft_consensus.cc:729] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677127814314770b3370764d0ff9d06, State: Initialized, Role: FOLLOWER
I20240501 20:17:32.216305 13550 consensus_queue.cc:260] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } }
I20240501 20:17:32.216881 13550 ts_tablet_manager.cc:1398] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:17:32.217772 27545 tablet_copy_service.cc:342] P 524f5e87a3ce40378bde76e667a4ece7: Request end of tablet copy session e677127814314770b3370764d0ff9d06-26cb2329d176441ca9127187db2d157a received from {username='slave'} at 127.14.106.195:33343
I20240501 20:17:32.217911 27545 tablet_copy_service.cc:433] P 524f5e87a3ce40378bde76e667a4ece7: ending tablet copy session e677127814314770b3370764d0ff9d06-26cb2329d176441ca9127187db2d157a on tablet 26cb2329d176441ca9127187db2d157a with peer e677127814314770b3370764d0ff9d06
I20240501 20:17:32.245472 27442 maintenance_manager.cc:419] P e677127814314770b3370764d0ff9d06: Scheduling UndoDeltaBlockGCOp(26cb2329d176441ca9127187db2d157a): 4581 bytes on disk
I20240501 20:17:32.246119 27333 maintenance_manager.cc:643] P e677127814314770b3370764d0ff9d06: UndoDeltaBlockGCOp(26cb2329d176441ca9127187db2d157a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240501 20:17:32.342981 27071 consensus_peers.cc:485] T 4dc5794a67374cc388da2edc47cbb113 P 4d603ea5d17441b3bac562bf08ec83fe -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:17:32.443431 13556 raft_consensus.cc:482] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212de34e3aa24fa1be12c7421df985fc)
I20240501 20:17:32.443614 13556 raft_consensus.cc:504] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:32.445108 13556 leader_election.cc:288] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969), 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
W20240501 20:17:32.446023 29400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
W20240501 20:17:32.447337 29400 leader_election.cc:334] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
W20240501 20:17:32.447710 29403 leader_election.cc:334] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:32.447839 29403 leader_election.cc:302] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e957d29835544ac587917cc067fb7423; no voters: 212de34e3aa24fa1be12c7421df985fc, 565cade0058540a4aa18d7c7a25affbe
I20240501 20:17:32.448191 13556 raft_consensus.cc:2726] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:17:32.458948 29518 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8oH_XT/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 20:11:45.092525 29521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:45.093010 29521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:45.105756 29521 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:48.209287 29521 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:11:50.221477 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.246897 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.247237 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.249392 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.249588 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.249964 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.250116 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.250324 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.250505 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.250684 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:50.250808 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.251003 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:50.251121 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:50.251291 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:50.251415 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:50.841475 29521 rebalancer_tool.cc:1074] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:51.328438 29521 rebalancer_tool.cc:1074] tablet b5a20d47052f404986315ddb05284ad9: '4d603ea5d17441b3bac562bf08ec83fe' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:51.699983 29521 rebalancer_tool.cc:1074] tablet 8bf74c24f28546bb90e10002270e3814: 'd8b307156e1346bd9eecf655f49924c7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:52.141409 29521 rebalancer_tool.cc:1074] tablet b5427fbc8a244a69974fd8d0f5f3d08d: 'e677127814314770b3370764d0ff9d06' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:52.619477 29521 rebalancer_tool.cc:1074] tablet 57b4bd50bc8f4d50957bd0280224c420: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:53.163269 29521 rebalancer_tool.cc:1074] tablet f02a619bc1584d53ac902c8d8baff976: '4d603ea5d17441b3bac562bf08ec83fe' -> '565cade0058540a4aa18d7c7a25affbe' move scheduled
I20240501 20:11:53.625087 29521 rebalancer_tool.cc:1074] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move scheduled
I20240501 20:11:54.229701 29521 rebalancer_tool.cc:1074] tablet af693dc19f844b12973b000093021619: 'e677127814314770b3370764d0ff9d06' -> 'e957d29835544ac587917cc067fb7423' move scheduled
I20240501 20:11:55.048805 29521 rebalancer_tool.cc:1074] tablet e7f03e02d13f44d188e5eb2774771a0d: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move scheduled
I20240501 20:11:55.477146 29521 rebalancer_tool.cc:1084] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> '565cade0058540a4aa18d7c7a25affbe' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240501 20:11:55.477391 29521 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:56.791450 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:56.791623 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:56.791800 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:56.791913 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:11:56.792011 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:56.792174 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:56.792305 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:56.792446 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:56.792552 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
I20240501 20:11:56.903354 29521 rebalancer_tool.cc:1074] tablet 74c414c2eed045bda9d85fe5c2fea654: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:11:56.995060 29521 rebalancer_tool.cc:1074] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:11:57.169085 29521 rebalancer_tool.cc:1084] tablet 50bf9058852a4ea49733326057d5ff41: 'd8b307156e1346bd9eecf655f49924c7' -> 'e957d29835544ac587917cc067fb7423' move ignored: Illegal state: tablet 50bf9058852a4ea49733326057d5ff41: replica e957d29835544ac587917cc067fb7423 is already present
I20240501 20:11:57.169267 29521 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:11:58.648684 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:58.648947 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:58.649142 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:58.649277 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:58.649417 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:58.649536 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc800bf02dcd4a3d8f1c164f02b5fbe5: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server e957d29835544ac587917cc067fb7423
W20240501 20:11:58.649657 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:58.649762 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:58.649899 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dd89cc8322c34fb1a2bd9dde8dbd37a2: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
W20240501 20:11:58.650020 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server d56cbcb06b9e418cb617de6a34b946b0
W20240501 20:11:58.650142 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 565e816257734920a19d1af631224041: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:11:58.650231 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server d8b307156e1346bd9eecf655f49924c7 to server 212de34e3aa24fa1be12c7421df985fc
I20240501 20:11:58.976830 29521 rebalancer_tool.cc:1074] tablet 553d443dfb344b2690c260d207901ff1: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:11:59.052089 29521 rebalancer_tool.cc:1084] tablet d2160f17e62749dbb41405a895516b06: '524f5e87a3ce40378bde76e667a4ece7' -> '565cade0058540a4aa18d7c7a25affbe' move ignored: Illegal state: tablet d2160f17e62749dbb41405a895516b06: replica 565cade0058540a4aa18d7c7a25affbe is already present
I20240501 20:11:59.052259 29521 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:12:00.896785 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:12:00.897065 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:00.897444 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bd611c8e8104400d8e0a2484f811bbe7: could not find any suitable replica to move from server e677127814314770b3370764d0ff9d06 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:00.897593 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e80affb3a7b40bda2dfa097a6c40afa: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:12:00.897753 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c64f22861534f1d84ad1e2a9970ff4a: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:00.897888 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server e957d29835544ac587917cc067fb7423
I20240501 20:12:01.004472 29521 rebalancer_tool.cc:1084] tablet 17e69dbd245e4b8db397aaffc9145c9f: 'd8b307156e1346bd9eecf655f49924c7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move ignored: Illegal state: tablet 17e69dbd245e4b8db397aaffc9145c9f: replica d56cbcb06b9e418cb617de6a34b946b0 is already present
I20240501 20:12:01.004658 29521 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:12:02.356309 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa6385b5bce14289828f8b7da259c90c: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 212de34e3aa24fa1be12c7421df985fc
W20240501 20:12:02.356539 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff19b427845646c9a01ebd69c335a06b: could not find any suitable replica to move from server 4d603ea5d17441b3bac562bf08ec83fe to server 565cade0058540a4aa18d7c7a25affbe
W20240501 20:12:02.356757 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92199738005a460797ae03d52d012acf: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server e957d29835544ac587917cc067fb7423
W20240501 20:12:02.357307 29521 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb0ab861e4ae485c8f71dd1c7e9f1022: could not find any suitable replica to move from server 524f5e87a3ce40378bde76e667a4ece7 to server 565cade0058540a4aa18d7c7a25affbe
I20240501 20:12:02.684700 29521 rebalancer_tool.cc:1074] tablet ebcdab26b4ed45f596282d4805d1bb1f: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:02.746261 29521 rebalancer_tool.cc:1074] tablet 4dc5794a67374cc388da2edc47cbb113: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:05.106441 29521 rebalancer_tool.cc:1123] tablet 4dc5794a67374cc388da2edc47cbb113: 'e677127814314770b3370764d0ff9d06' -> '' move is abandoned: Not found: Server with UUID e677127814314770b3370764d0ff9d06 not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:12:12.218955 29521 rebalancer_tool.cc:1136] tablet 74c414c2eed045bda9d85fe5c2fea654: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:18.726130 29521 rebalancer_tool.cc:1136] tablet 7bb13c1dce8640b28063daa0132d2896: 'd8b307156e1346bd9eecf655f49924c7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:19.062925 29521 rebalancer_tool.cc:1136] tablet b5a20d47052f404986315ddb05284ad9: '4d603ea5d17441b3bac562bf08ec83fe' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:19.165612 29521 rebalancer_tool.cc:1136] tablet b5427fbc8a244a69974fd8d0f5f3d08d: 'e677127814314770b3370764d0ff9d06' -> 'e957d29835544ac587917cc067fb7423' move completed: OK
I20240501 20:12:23.140684 29521 rebalancer_tool.cc:1123] tablet 78fcaa70e916442ea9c67cc762c8db6d: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move is abandoned: Not found: Server with UUID 524f5e87a3ce40378bde76e667a4ece7 not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "d56cbcb06b9e418cb617de6a34b946b0" member_type: VOTER last_known_addr { host: "127.14.106.198" port: 46843 } }
I20240501 20:12:25.178311 29521 rebalancer_tool.cc:1136] tablet 57b4bd50bc8f4d50957bd0280224c420: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:25.256325 29521 rebalancer_tool.cc:1074] tablet c0e88937e1ec49bfbaffe535e445e40a: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:27.106936 29521 rebalancer_tool.cc:1136] tablet ebcdab26b4ed45f596282d4805d1bb1f: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:27.187044 29521 rebalancer_tool.cc:1136] tablet 553d443dfb344b2690c260d207901ff1: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:27.320578 29521 rebalancer_tool.cc:1136] tablet 6c836d9afa0f439b978bc2fe5b5e499d: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:27.446164 29521 rebalancer_tool.cc:1136] tablet e7f03e02d13f44d188e5eb2774771a0d: '524f5e87a3ce40378bde76e667a4ece7' -> '212de34e3aa24fa1be12c7421df985fc' move completed: OK
I20240501 20:12:27.609534 29521 rebalancer_tool.cc:1136] tablet af693dc19f844b12973b000093021619: 'e677127814314770b3370764d0ff9d06' -> 'e957d29835544ac587917cc067fb7423' move completed: OK
I20240501 20:12:27.693779 29521 rebalancer_tool.cc:1136] tablet f02a619bc1584d53ac902c8d8baff976: '4d603ea5d17441b3bac562bf08ec83fe' -> '565cade0058540a4aa18d7c7a25affbe' move completed: OK
I20240501 20:12:27.779361 29521 rebalancer_tool.cc:1136] tablet 8bf74c24f28546bb90e10002270e3814: 'd8b307156e1346bd9eecf655f49924c7' -> 'd56cbcb06b9e418cb617de6a34b946b0' move completed: OK
I20240501 20:12:27.828205 29521 rebalancer_tool.cc:1074] tablet e5b9f1a087114c82b010bd10e4f676c1: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:27.892514 29521 rebalancer_tool.cc:1074] tablet d2160f17e62749dbb41405a895516b06: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:27.965561 29521 rebalancer_tool.cc:1074] tablet f034a679870b44caa238f04a8b49d302: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:28.036168 29521 rebalancer_tool.cc:1136] tablet f034a679870b44caa238f04a8b49d302: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:29.237571 29521 rebalancer_tool.cc:1136] tablet d2160f17e62749dbb41405a895516b06: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:29.286020 29521 rebalancer_tool.cc:1136] tablet c0e88937e1ec49bfbaffe535e445e40a: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:29.375720 29521 rebalancer_tool.cc:1074] tablet 0c836bb145a54160966c475e1490da38: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:29.421629 29521 rebalancer_tool.cc:1074] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move scheduled
I20240501 20:12:29.485307 29521 rebalancer_tool.cc:1074] tablet 00e3777032eb4d9c80f47855e5f9f87a: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:29.577172 29521 rebalancer_tool.cc:1136] tablet 00e3777032eb4d9c80f47855e5f9f87a: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:29.774441 29521 rebalancer_tool.cc:1136] tablet 9721fc9afa4741b78823e293514e6265: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
I20240501 20:12:29.891777 29521 rebalancer_tool.cc:1136] tablet 0c836bb145a54160966c475e1490da38: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:31.635402 29521 rebalancer_tool.cc:1074] tablet c619a2235c9248eeb6b2e1d78583d1ba: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:31.648197 29521 rebalancer_tool.cc:1074] tablet c2432c8e8e8847fa977055be919e79d6: '524f5e87a3ce40378bde76e667a4ece7' -> '' move scheduled
I20240501 20:12:31.794435 29521 rebalancer_tool.cc:1074] tablet 3f8902b7b0b649edb10f362713ad36fe: 'e677127814314770b3370764d0ff9d06' -> '' move scheduled
I20240501 20:12:32.030606 29521 rebalancer_tool.cc:1136] tablet 3f8902b7b0b649edb10f362713ad36fe: 'e677127814314770b3370764d0ff9d06' -> '' move completed: OK
I20240501 20:12:32.156651 29521 rebalancer_tool.cc:1136] tablet c2432c8e8e8847fa977055be919e79d6: '524f5e87a3ce40378bde76e667a4ece7' -> '' move completed: OK
I20240501 20:12:32.223868 29521 rebalancer_tool.cc:1136] tablet c619a2235c9248eeb6b2e1d78583d1ba: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
I20240501 20:12:33.338209 29521 rebalancer_tool.cc:1074] tablet 26cb2329d176441ca9127187db2d157a: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:33.446560 29521 rebalancer_tool.cc:1074] tablet 944d354124664bc0870f251d8a0831c5: 'd8b307156e1346bd9eecf655f49924c7' -> '' move scheduled
I20240501 20:12:33.488169 29521 rebalancer_tool.cc:1136] tablet 944d354124664bc0870f251d8a0831c5: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
W20240501 20:14:55.191475 24226 meta_cache.cc:1261] Time spent looking up entry by key: real 0.069s user 0.000s sys 0.000s
W20240501 20:16:31.643817 5523 meta_cache.cc:1097] Time spent processing table locations response: real 0.055s user 0.001s sys 0.000s
W20240501 20:16:31.644418 5523 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.14.106.254:34283, user_credentials={real_user=slave}} blocked reactor thread for 55496us
W20240501 20:17:29.747567 13179 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:17:29.511476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.106.197:40723 (local address 127.0.0.1:51420)
0501 20:17:29.511730 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0501 20:17:29.511755 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 20:17:29.511889 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:17:29.746264 (+234375us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114}
I20240501 20:17:30.454538 29521 rebalancer_tool.cc:1136] tablet e5b9f1a087114c82b010bd10e4f676c1: '4d603ea5d17441b3bac562bf08ec83fe' -> '' move completed: OK
W20240501 20:17:31.551980 13421 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:17:31.388534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.106.199:32969 (local address 127.0.0.1:50942)
0501 20:17:31.388752 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0501 20:17:31.388775 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 20:17:31.388902 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:17:31.551769 (+162867us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":106}
I20240501 20:17:31.826149 29521 rebalancer_tool.cc:1136] tablet 26cb2329d176441ca9127187db2d157a: 'd8b307156e1346bd9eecf655f49924c7' -> '' move completed: OK
Illegal state: tablet server 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): unacceptable health status UNAVAILABLE
E20240501 20:17:32.459717 14763 external_mini_cluster.cc:943] Process with UUID 565cade0058540a4aa18d7c7a25affbe has crashed
E20240501 20:17:32.459844 14763 external_mini_cluster.cc:943] Process with UUID d56cbcb06b9e418cb617de6a34b946b0 has crashed
E20240501 20:17:32.459924 14763 external_mini_cluster.cc:943] Process with UUID 212de34e3aa24fa1be12c7421df985fc has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 3
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240501 20:17:32.461457 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27053
I20240501 20:17:32.473675 27170 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:17:32.505986 27202 consensus_peers.cc:462] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
W20240501 20:17:32.572264 27329 consensus_peers.cc:462] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
W20240501 20:17:32.614238 26979 catalog_manager.cc:4573] TS d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 00e3777032eb4d9c80f47855e5f9f87a: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
I20240501 20:17:32.645293 27394 raft_consensus.cc:1203] T 26cb2329d176441ca9127187db2d157a P e677127814314770b3370764d0ff9d06 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 20:17:32.873172 13428 raft_consensus.cc:482] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212de34e3aa24fa1be12c7421df985fc)
I20240501 20:17:32.873428 13428 raft_consensus.cc:504] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:32.884068 13428 leader_election.cc:288] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723), 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
W20240501 20:17:32.886010 27071 leader_election.cc:334] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
W20240501 20:17:32.886843 27068 leader_election.cc:334] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
I20240501 20:17:32.886991 27068 leader_election.cc:302] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d603ea5d17441b3bac562bf08ec83fe; no voters: 212de34e3aa24fa1be12c7421df985fc, 565cade0058540a4aa18d7c7a25affbe
I20240501 20:17:32.887306 13562 raft_consensus.cc:2726] T 553d443dfb344b2690c260d207901ff1 P 4d603ea5d17441b3bac562bf08ec83fe [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:17:32.888178 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 27053
W20240501 20:17:32.905481 27202 consensus_peers.cc:462] T e5b9f1a087114c82b010bd10e4f676c1 P d8b307156e1346bd9eecf655f49924c7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
W20240501 20:17:32.948273 29403 connection.cc:535] server connection from 127.14.106.193:36217 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:32.948915 27202 connection.cc:535] server connection from 127.14.106.193:37475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:17:32.949143 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27184
I20240501 20:17:32.955132 27301 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:17:32.973942 27461 consensus_peers.cc:485] T 7bb13c1dce8640b28063daa0132d2896 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Couldn't send request to peer 4d603ea5d17441b3bac562bf08ec83fe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:32.987565 27329 consensus_peers.cc:462] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
W20240501 20:17:33.104601 27331 consensus_peers.cc:485] T d2160f17e62749dbb41405a895516b06 P e677127814314770b3370764d0ff9d06 -> Peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Couldn't send request to peer 4d603ea5d17441b3bac562bf08ec83fe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:33.321703 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 27184
W20240501 20:17:33.380400 27330 connection.cc:535] server connection from 127.14.106.194:38415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:33.380883 27331 connection.cc:535] client connection to 127.14.106.194:46281 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:17:33.381194 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27314
W20240501 20:17:33.381886 27461 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Couldn't send request to peer d8b307156e1346bd9eecf655f49924c7. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.194:46281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:33.382014 27331 consensus_peers.cc:485] T ebcdab26b4ed45f596282d4805d1bb1f P e677127814314770b3370764d0ff9d06 -> Peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Couldn't send request to peer d8b307156e1346bd9eecf655f49924c7. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.194:46281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:33.382143 27331 consensus_peers.cc:485] T 944d354124664bc0870f251d8a0831c5 P e677127814314770b3370764d0ff9d06 -> Peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Couldn't send request to peer d8b307156e1346bd9eecf655f49924c7. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.194:46281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:17:33.382220 27331 consensus_peers.cc:485] T 17e69dbd245e4b8db397aaffc9145c9f P e677127814314770b3370764d0ff9d06 -> Peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Couldn't send request to peer d8b307156e1346bd9eecf655f49924c7. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.194:46281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:33.387393 27430 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:17:33.648234 27459 consensus_peers.cc:485] T 00e3777032eb4d9c80f47855e5f9f87a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:33.714920 27461 consensus_peers.cc:485] T 9721fc9afa4741b78823e293514e6265 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Couldn't send request to peer d8b307156e1346bd9eecf655f49924c7. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.194:46281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:33.765270 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 27314
W20240501 20:17:33.767422 26979 catalog_manager.cc:4573] TS d56cbcb06b9e418cb617de6a34b946b0 (127.14.106.198:46843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 57b4bd50bc8f4d50957bd0280224c420: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111)
W20240501 20:17:33.821987 29403 connection.cc:535] server connection from 127.14.106.195:38865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:17:33.822206 26979 connection.cc:535] server connection from 127.14.106.195:46823 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:17:33.822533 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27444
W20240501 20:17:33.824106 27462 consensus_peers.cc:485] T 26cb2329d176441ca9127187db2d157a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Couldn't send request to peer e677127814314770b3370764d0ff9d06. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.195:35443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:33.828102 27562 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:33.831530 13588 raft_consensus.cc:482] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d603ea5d17441b3bac562bf08ec83fe)
I20240501 20:17:33.831804 13588 raft_consensus.cc:504] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:33.832710 13588 leader_election.cc:288] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196:43861)
W20240501 20:17:33.834242 29402 leader_election.cc:334] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111)
I20240501 20:17:33.955034 13588 raft_consensus.cc:482] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d603ea5d17441b3bac562bf08ec83fe)
I20240501 20:17:33.955243 13588 raft_consensus.cc:504] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:33.955974 13588 leader_election.cc:288] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e677127814314770b3370764d0ff9d06 (127.14.106.195:35443)
W20240501 20:17:33.957336 29402 leader_election.cc:334] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111)
W20240501 20:17:33.957916 29403 leader_election.cc:334] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Network error: Client connection negotiation failed: client connection to 127.14.106.195:35443: connect: Connection refused (error 111)
I20240501 20:17:33.958035 29403 leader_election.cc:302] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e957d29835544ac587917cc067fb7423; no voters: 4d603ea5d17441b3bac562bf08ec83fe, e677127814314770b3370764d0ff9d06
I20240501 20:17:33.958283 13588 raft_consensus.cc:2726] T 309e90bbd9f140a8beca9b3da3f75db4 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:17:34.207038 27459 consensus_peers.cc:485] T c0e88937e1ec49bfbaffe535e445e40a P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:34.207267 27459 consensus_peers.cc:485] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Couldn't send request to peer 212de34e3aa24fa1be12c7421df985fc. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:17:34.209494 27462 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Couldn't send request to peer e677127814314770b3370764d0ff9d06. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.195:35443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:17:34.209764 27526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "e957d29835544ac587917cc067fb7423" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "524f5e87a3ce40378bde76e667a4ece7" is_pre_election: true
I20240501 20:17:34.210505 29401 leader_election.cc:302] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e957d29835544ac587917cc067fb7423; no voters: 4d603ea5d17441b3bac562bf08ec83fe, 524f5e87a3ce40378bde76e667a4ece7
I20240501 20:17:34.210953 13594 raft_consensus.cc:482] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d603ea5d17441b3bac562bf08ec83fe)
I20240501 20:17:34.211025 13588 raft_consensus.cc:2726] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:17:34.211107 13594 raft_consensus.cc:504] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:34.211725 13594 leader_election.cc:288] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
W20240501 20:17:34.252395 27459 leader_election.cc:334] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
W20240501 20:17:34.257524 27461 leader_election.cc:334] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111)
I20240501 20:17:34.258103 27461 leader_election.cc:302] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7; no voters: 212de34e3aa24fa1be12c7421df985fc, 4d603ea5d17441b3bac562bf08ec83fe
I20240501 20:17:34.262431 13594 raft_consensus.cc:2726] T e7f03e02d13f44d188e5eb2774771a0d P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:17:34.263882 27462 consensus_peers.cc:485] T f02a619bc1584d53ac902c8d8baff976 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:17:34.378225 13594 raft_consensus.cc:482] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d603ea5d17441b3bac562bf08ec83fe)
I20240501 20:17:34.378706 13588 raft_consensus.cc:482] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d603ea5d17441b3bac562bf08ec83fe)
I20240501 20:17:34.378649 13594 raft_consensus.cc:504] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:34.379051 13588 raft_consensus.cc:504] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } }
I20240501 20:17:34.380342 13588 leader_election.cc:288] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
I20240501 20:17:34.380578 13588 raft_consensus.cc:482] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d603ea5d17441b3bac562bf08ec83fe)
I20240501 20:17:34.380786 13588 raft_consensus.cc:504] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:34.381191 13594 leader_election.cc:288] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e957d29835544ac587917cc067fb7423 (127.14.106.200:44949)
I20240501 20:17:34.381953 13588 leader_election.cc:288] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
I20240501 20:17:34.384814 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e957d29835544ac587917cc067fb7423" is_pre_election: true
I20240501 20:17:34.385169 29464 raft_consensus.cc:2454] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 1.
W20240501 20:17:34.385221 27461 leader_election.cc:334] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111)
W20240501 20:17:34.387480 29403 leader_election.cc:334] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Network error: Client connection negotiation failed: client connection to 127.14.106.195:35443: connect: Connection refused (error 111)
W20240501 20:17:34.387771 29403 leader_election.cc:334] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
W20240501 20:17:34.390682 29400 leader_election.cc:334] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
I20240501 20:17:34.391008 29400 leader_election.cc:302] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e957d29835544ac587917cc067fb7423; no voters: 212de34e3aa24fa1be12c7421df985fc, e677127814314770b3370764d0ff9d06
I20240501 20:17:34.391487 13588 raft_consensus.cc:2726] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:17:34.391641 29402 leader_election.cc:334] T 50bf9058852a4ea49733326057d5ff41 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111)
W20240501 20:17:34.392133 29402 leader_election.cc:334] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111)
I20240501 20:17:34.392300 29402 leader_election.cc:302] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e957d29835544ac587917cc067fb7423; no voters: 4d603ea5d17441b3bac562bf08ec83fe, 565cade0058540a4aa18d7c7a25affbe
I20240501 20:17:34.392642 13588 raft_consensus.cc:2726] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:17:34.392715 29402 leader_election.cc:334] T b5427fbc8a244a69974fd8d0f5f3d08d P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Network error: Client connection negotiation failed: client connection to 127.14.106.194:46281: connect: Connection refused (error 111)
I20240501 20:17:34.393651 27461 leader_election.cc:302] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e957d29835544ac587917cc067fb7423; no voters: 4d603ea5d17441b3bac562bf08ec83fe
I20240501 20:17:34.394729 13594 raft_consensus.cc:2781] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:17:34.394902 13594 raft_consensus.cc:482] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d603ea5d17441b3bac562bf08ec83fe)
I20240501 20:17:34.395090 13594 raft_consensus.cc:3037] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:17:34.408340 13594 raft_consensus.cc:504] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:34.410414 13594 leader_election.cc:288] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 2 election: Requested vote from peers 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293), e957d29835544ac587917cc067fb7423 (127.14.106.200:44949)
I20240501 20:17:34.411984 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf74c24f28546bb90e10002270e3814" candidate_uuid: "524f5e87a3ce40378bde76e667a4ece7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e957d29835544ac587917cc067fb7423"
I20240501 20:17:34.412320 29464 raft_consensus.cc:3037] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:17:34.413678 27461 leader_election.cc:334] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193:46293): Network error: Client connection negotiation failed: client connection to 127.14.106.193:46293: connect: Connection refused (error 111)
I20240501 20:17:34.414652 13599 raft_consensus.cc:482] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8b307156e1346bd9eecf655f49924c7)
I20240501 20:17:34.414825 13599 raft_consensus.cc:504] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } }
I20240501 20:17:34.415771 13599 leader_election.cc:288] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281), 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
W20240501 20:17:34.417135 29403 leader_election.cc:334] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
W20240501 20:17:34.417433 29402 leader_election.cc:334] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Network error: Client connection negotiation failed: client connection to 127.14.106.194:46281: connect: Connection refused (error 111)
I20240501 20:17:34.417623 29402 leader_election.cc:302] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e957d29835544ac587917cc067fb7423; no voters: 565cade0058540a4aa18d7c7a25affbe, d8b307156e1346bd9eecf655f49924c7
I20240501 20:17:34.417979 13599 raft_consensus.cc:2726] T e5b9f1a087114c82b010bd10e4f676c1 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:17:34.418231 29464 raft_consensus.cc:2454] T 8bf74c24f28546bb90e10002270e3814 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524f5e87a3ce40378bde76e667a4ece7 in term 2.
I20240501 20:17:34.419214 27461 leader_election.cc:302] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 524f5e87a3ce40378bde76e667a4ece7, e957d29835544ac587917cc067fb7423; no voters: 4d603ea5d17441b3bac562bf08ec83fe
I20240501 20:17:34.419838 13594 raft_consensus.cc:2781] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:17:34.420159 13594 raft_consensus.cc:686] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [term 2 LEADER]: Becoming Leader. State: Replica: 524f5e87a3ce40378bde76e667a4ece7, State: Running, Role: LEADER
I20240501 20:17:34.422071 13594 consensus_queue.cc:237] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } }
I20240501 20:17:34.428787 26990 catalog_manager.cc:5484] T 8bf74c24f28546bb90e10002270e3814 P 524f5e87a3ce40378bde76e667a4ece7 reported cstate change: term changed from 1 to 2, leader changed from 4d603ea5d17441b3bac562bf08ec83fe (127.14.106.193) to 524f5e87a3ce40378bde76e667a4ece7 (127.14.106.196). New cstate: current_term: 2 leader_uuid: "524f5e87a3ce40378bde76e667a4ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d603ea5d17441b3bac562bf08ec83fe" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46293 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } } peers { permanent_uuid: "524f5e87a3ce40378bde76e667a4ece7" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 43861 } } }
I20240501 20:17:34.534541 13599 raft_consensus.cc:482] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212de34e3aa24fa1be12c7421df985fc)
I20240501 20:17:34.534781 13599 raft_consensus.cc:504] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "565cade0058540a4aa18d7c7a25affbe" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 40723 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:34.535503 13599 leader_election.cc:288] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969), 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723)
W20240501 20:17:34.536900 29400 leader_election.cc:334] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969): Network error: Client connection negotiation failed: client connection to 127.14.106.199:32969: connect: Connection refused (error 111)
W20240501 20:17:34.537129 29403 leader_election.cc:334] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111)
I20240501 20:17:34.537340 29403 leader_election.cc:302] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e957d29835544ac587917cc067fb7423; no voters: 212de34e3aa24fa1be12c7421df985fc, 565cade0058540a4aa18d7c7a25affbe
I20240501 20:17:34.537770 13599 raft_consensus.cc:2726] T f034a679870b44caa238f04a8b49d302 P e957d29835544ac587917cc067fb7423 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:17:34.668715 27462 consensus_peers.cc:485] T 0c836bb145a54160966c475e1490da38 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:17:34.762727 27462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 20:17:34.764911 27462 consensus_peers.cc:485] T af693dc19f844b12973b000093021619 P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:17:34.815833 27462 consensus_peers.cc:485] T 6c836d9afa0f439b978bc2fe5b5e499d P 524f5e87a3ce40378bde76e667a4ece7 -> Peer 565cade0058540a4aa18d7c7a25affbe (127.14.106.197:40723): Couldn't send request to peer 565cade0058540a4aa18d7c7a25affbe. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.197:40723: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:17:34.823833 14763 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 27444: Timed out: CheckLeaks RPC to 127.14.106.196:43861 timed out after 1.000s (SENT)
I20240501 20:17:34.824174 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 27444
I20240501 20:17:34.841539 13599 raft_consensus.cc:482] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e677127814314770b3370764d0ff9d06)
I20240501 20:17:34.841759 13599 raft_consensus.cc:504] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "d8b307156e1346bd9eecf655f49924c7" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 46281 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:34.842505 13599 leader_election.cc:288] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281)
W20240501 20:17:34.844092 29403 leader_election.cc:334] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e677127814314770b3370764d0ff9d06 (127.14.106.195:35443): Network error: Client connection negotiation failed: client connection to 127.14.106.195:35443: connect: Connection refused (error 111)
W20240501 20:17:34.844290 29402 leader_election.cc:334] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d8b307156e1346bd9eecf655f49924c7 (127.14.106.194:46281): Network error: Client connection negotiation failed: client connection to 127.14.106.194:46281: connect: Connection refused (error 111)
I20240501 20:17:34.844398 29402 leader_election.cc:302] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e957d29835544ac587917cc067fb7423; no voters: d8b307156e1346bd9eecf655f49924c7, e677127814314770b3370764d0ff9d06
I20240501 20:17:34.844651 13599 raft_consensus.cc:2726] T ebcdab26b4ed45f596282d4805d1bb1f P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:17:34.879853 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 29384
I20240501 20:17:34.891498 29504 generic_service.cc:196] Checking for leaks (request via RPC)
LLVM ERROR: IO failure on output stream: Broken pipe
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
/tmp/dist-test-task8oH_XT/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcdfce)[0x5583c8034fce]
/tmp/dist-test-task8oH_XT/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcc27d)[0x5583c803327d]
/tmp/dist-test-task8oH_XT/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcc3d3)[0x5583c80333d3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f61d88af980]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f61d798dfb7]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f61d798f921]
/tmp/dist-test-task8oH_XT/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xa575c)[0x5583c800c75c]
/tmp/dist-test-task8oH_XT/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xa58e0)[0x5583c800c8e0]
/tmp/dist-test-task8oH_XT/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xbcc73)[0x5583c8023c73]
/lib/x86_64-linux-gnu/libc.so.6(+0x43161)[0x7f61d7992161]
/lib/x86_64-linux-gnu/libc.so.6(+0x4325a)[0x7f61d799225a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xee)[0x7f61d7970bfe]
/tmp/dist-test-task8oH_XT/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0x8f0ba)[0x5583c7ff60ba]
I20240501 20:17:35.314389 13599 raft_consensus.cc:482] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e677127814314770b3370764d0ff9d06)
I20240501 20:17:35.314599 13599 raft_consensus.cc:504] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e677127814314770b3370764d0ff9d06" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 35443 } } peers { permanent_uuid: "212de34e3aa24fa1be12c7421df985fc" member_type: VOTER last_known_addr { host: "127.14.106.199" port: 32969 } } peers { permanent_uuid: "e957d29835544ac587917cc067fb7423" member_type: VOTER last_known_addr { host: "127.14.106.200" port: 44949 } }
I20240501 20:17:35.315405 13599 leader_election.cc:288] T 57b4bd50bc8f4d50957bd0280224c420 P e957d29835544ac587917cc067fb7423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e677127814314770b3370764d0ff9d06 (127.14.106.195:35443), 212de34e3aa24fa1be12c7421df985fc (127.14.106.199:32969)
I20240501 20:17:35.315546 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 29384
I20240501 20:17:35.373710 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 26963
I20240501 20:17:35.380647 27020 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:35.739472 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 26963
W20240501 20:17:35.739435 26979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.198:46843: connect: Connection refused (error 111) [suppressed 30 similar messages]
2024-05-01T20:17:35Z chronyd exiting
I20240501 20:17:35.811997 14763 test_util.cc:175] -----------------------------------------------
I20240501 20:17:35.812093 14763 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714594103620019-14763-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (375813 ms)
[----------] 1 test from ConcurrentRebalancersTest (375813 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240501 20:17:35.814301 14763 test_util.cc:268] Using random seed: 761578112
I20240501 20:17:35.816035 14763 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:17:35.816098 14763 ts_itest-base.cc:113] --------------
I20240501 20:17:35.816151 14763 ts_itest-base.cc:114] 4 tablet servers
I20240501 20:17:35.816211 14763 ts_itest-base.cc:115] 1 replicas per TS
I20240501 20:17:35.816272 14763 ts_itest-base.cc:116] --------------
2024-05-01T20:17:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:17:35Z Disabled control of system clock
I20240501 20:17:35.868788 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:35537
--webserver_interface=127.14.106.254
--webserver_port=0
--builtin_ntp_servers=127.14.106.212:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.106.254:35537
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240501 20:17:36.219156 13620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:36.219465 13620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:36.219571 13620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:36.227031 13620 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:17:36.227144 13620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:17:36.227200 13620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:36.227257 13620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:17:36.227303 13620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:17:36.236217 13620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.106.254:35537
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:36.237555 13620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:36.239457 13620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:36.250516 13625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:36.251024 13626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:36.251436 13628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:36.252038 13620 server_base.cc:1034] running on GCE node
I20240501 20:17:36.252718 13620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:36.254012 13620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:36.255205 13620 hybrid_clock.cc:648] HybridClock initialized: now 1714594656255161 us; error 41 us; skew 500 ppm
I20240501 20:17:36.255515 13620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:36.257651 13620 webserver.cc:460] Webserver started at http://127.14.106.254:41079/ using document root <none> and password file <none>
I20240501 20:17:36.258345 13620 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:36.258469 13620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:36.258821 13620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:36.261348 13620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "aa0a736e8ebf4701a363da1d56b165b7"
format_stamp: "Formatted at 2024-05-01 20:17:36 on dist-test-slave-pg4x"
I20240501 20:17:36.262107 13620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "aa0a736e8ebf4701a363da1d56b165b7"
format_stamp: "Formatted at 2024-05-01 20:17:36 on dist-test-slave-pg4x"
I20240501 20:17:36.267110 13620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20240501 20:17:36.270251 13634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:36.271500 13620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:36.271662 13620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
uuid: "aa0a736e8ebf4701a363da1d56b165b7"
format_stamp: "Formatted at 2024-05-01 20:17:36 on dist-test-slave-pg4x"
I20240501 20:17:36.271857 13620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:36.312084 13620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:36.313026 13620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:36.313341 13620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:36.331681 13620 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.254:35537
I20240501 20:17:36.331698 13685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.254:35537 every 8 connection(s)
I20240501 20:17:36.333065 13620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:17:36.336930 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:36.340543 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 13620
I20240501 20:17:36.340793 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:17:36.347905 13686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7: Bootstrap starting.
I20240501 20:17:36.351492 13686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:36.352694 13686 log.cc:824] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:17:36.355636 13686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7: No bootstrap required, opened a new log
I20240501 20:17:36.362012 13686 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0a736e8ebf4701a363da1d56b165b7" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 35537 } }
I20240501 20:17:36.362370 13686 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:36.362481 13686 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0a736e8ebf4701a363da1d56b165b7, State: Initialized, Role: FOLLOWER
I20240501 20:17:36.363123 13686 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0a736e8ebf4701a363da1d56b165b7" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 35537 } }
I20240501 20:17:36.363337 13686 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:36.363469 13686 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:36.363622 13686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:36.365847 13686 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0a736e8ebf4701a363da1d56b165b7" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 35537 } }
I20240501 20:17:36.366326 13686 leader_election.cc:302] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa0a736e8ebf4701a363da1d56b165b7; no voters:
I20240501 20:17:36.366935 13686 leader_election.cc:288] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:36.367069 13691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:36.367709 13691 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [term 1 LEADER]: Becoming Leader. State: Replica: aa0a736e8ebf4701a363da1d56b165b7, State: Running, Role: LEADER
I20240501 20:17:36.368254 13691 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0a736e8ebf4701a363da1d56b165b7" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 35537 } }
I20240501 20:17:36.368889 13686 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:17:36.370952 13693 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0a736e8ebf4701a363da1d56b165b7. Latest consensus state: current_term: 1 leader_uuid: "aa0a736e8ebf4701a363da1d56b165b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0a736e8ebf4701a363da1d56b165b7" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 35537 } } }
I20240501 20:17:36.371376 13693 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [sys.catalog]: This master's current role is: LEADER
I20240501 20:17:36.377490 13699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:17:36.376770 13692 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa0a736e8ebf4701a363da1d56b165b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0a736e8ebf4701a363da1d56b165b7" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 35537 } } }
I20240501 20:17:36.377732 13692 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7 [sys.catalog]: This master's current role is: LEADER
I20240501 20:17:36.382874 13699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:17:36.391201 13699 catalog_manager.cc:1347] Generated new cluster ID: b4b724de845d4891a139265564daedee
I20240501 20:17:36.391310 13699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:17:36.424506 13699 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:17:36.425998 13699 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:17:36.433473 13699 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aa0a736e8ebf4701a363da1d56b165b7: Generated new TSK 0
I20240501 20:17:36.434262 13699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:17:36.440399 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.193:0
--local_ip_for_outbound_sockets=127.14.106.193
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:35537
--builtin_ntp_servers=127.14.106.212:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:17:36.801018 13710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:36.801323 13710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:36.801470 13710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:36.809140 13710 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:17:36.809347 13710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:36.809660 13710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.193
I20240501 20:17:36.819200 13710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:35537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.193
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:36.820478 13710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:36.822177 13710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:36.833968 13715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:36.834331 13716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:36.834935 13718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:36.835464 13710 server_base.cc:1034] running on GCE node
I20240501 20:17:36.836005 13710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:36.836913 13710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:36.838116 13710 hybrid_clock.cc:648] HybridClock initialized: now 1714594656838061 us; error 51 us; skew 500 ppm
I20240501 20:17:36.838430 13710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:36.840636 13710 webserver.cc:460] Webserver started at http://127.14.106.193:44085/ using document root <none> and password file <none>
I20240501 20:17:36.841364 13710 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:36.841497 13710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:36.841820 13710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:36.844521 13710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6588a03404be4ed19b41e92920ca66f7"
format_stamp: "Formatted at 2024-05-01 20:17:36 on dist-test-slave-pg4x"
I20240501 20:17:36.845254 13710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6588a03404be4ed19b41e92920ca66f7"
format_stamp: "Formatted at 2024-05-01 20:17:36 on dist-test-slave-pg4x"
I20240501 20:17:36.850740 13710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 20:17:36.854274 13724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:36.855679 13710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:36.855840 13710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6588a03404be4ed19b41e92920ca66f7"
format_stamp: "Formatted at 2024-05-01 20:17:36 on dist-test-slave-pg4x"
I20240501 20:17:36.856045 13710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:36.879194 13710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:36.880055 13710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:36.880324 13710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:36.881254 13710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:36.882782 13710 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:36.882882 13710 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:36.883000 13710 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:36.883072 13710 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:36.915357 13710 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.193:34519
I20240501 20:17:36.915526 13837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.193:34519 every 8 connection(s)
I20240501 20:17:36.916711 13710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:17:36.924873 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 13710
I20240501 20:17:36.925089 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:17:36.928836 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.194:0
--local_ip_for_outbound_sockets=127.14.106.194
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:35537
--builtin_ntp_servers=127.14.106.212:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:17:36.929893 13838 heartbeater.cc:347] Connected to a master server at 127.14.106.254:35537
I20240501 20:17:36.930163 13838 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:36.930903 13838 heartbeater.cc:511] Master 127.14.106.254:35537 requested a full tablet report, sending...
I20240501 20:17:36.933075 13651 ts_manager.cc:194] Registered new tserver with Master: 6588a03404be4ed19b41e92920ca66f7 (127.14.106.193:34519)
I20240501 20:17:36.935230 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.193:59635
W20240501 20:17:37.272641 13842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:37.272984 13842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:37.273094 13842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:37.280581 13842 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:17:37.280696 13842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:37.280826 13842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.194
I20240501 20:17:37.289669 13842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:35537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.194
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:37.290935 13842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:37.292603 13842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:37.304759 13847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:37.305224 13848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:37.305727 13850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:37.306258 13842 server_base.cc:1034] running on GCE node
I20240501 20:17:37.306838 13842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:37.307679 13842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:37.308884 13842 hybrid_clock.cc:648] HybridClock initialized: now 1714594657308843 us; error 29 us; skew 500 ppm
I20240501 20:17:37.309197 13842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:37.311555 13842 webserver.cc:460] Webserver started at http://127.14.106.194:38475/ using document root <none> and password file <none>
I20240501 20:17:37.312254 13842 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:37.312376 13842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:37.312675 13842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:37.315308 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9e5726ef69254e618c4be6a1daabaf6c"
format_stamp: "Formatted at 2024-05-01 20:17:37 on dist-test-slave-pg4x"
I20240501 20:17:37.316056 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9e5726ef69254e618c4be6a1daabaf6c"
format_stamp: "Formatted at 2024-05-01 20:17:37 on dist-test-slave-pg4x"
I20240501 20:17:37.320933 13842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 20:17:37.324100 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:37.325500 13842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:17:37.325647 13842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9e5726ef69254e618c4be6a1daabaf6c"
format_stamp: "Formatted at 2024-05-01 20:17:37 on dist-test-slave-pg4x"
I20240501 20:17:37.325821 13842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:37.344444 13842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:37.345211 13842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:37.345468 13842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:37.346518 13842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:37.347955 13842 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:37.348057 13842 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:37.348183 13842 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:37.348254 13842 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:37.380906 13842 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.194:38669
I20240501 20:17:37.380955 13969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.194:38669 every 8 connection(s)
I20240501 20:17:37.382164 13842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:17:37.384742 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 13842
I20240501 20:17:37.384940 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:17:37.388288 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.195:0
--local_ip_for_outbound_sockets=127.14.106.195
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:35537
--builtin_ntp_servers=127.14.106.212:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:17:37.395442 13970 heartbeater.cc:347] Connected to a master server at 127.14.106.254:35537
I20240501 20:17:37.395745 13970 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:37.396468 13970 heartbeater.cc:511] Master 127.14.106.254:35537 requested a full tablet report, sending...
I20240501 20:17:37.397859 13651 ts_manager.cc:194] Registered new tserver with Master: 9e5726ef69254e618c4be6a1daabaf6c (127.14.106.194:38669)
I20240501 20:17:37.399085 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.194:34465
W20240501 20:17:37.727097 13973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:37.727432 13973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:37.727584 13973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:37.735139 13973 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:17:37.735292 13973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:37.735469 13973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.195
I20240501 20:17:37.744518 13973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:35537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.195
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:37.745847 13973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:37.747622 13973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:37.759738 13979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:37.760257 13978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:37.760633 13981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:37.761413 13973 server_base.cc:1034] running on GCE node
I20240501 20:17:37.761993 13973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:37.763016 13973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:37.764232 13973 hybrid_clock.cc:648] HybridClock initialized: now 1714594657764174 us; error 46 us; skew 500 ppm
I20240501 20:17:37.764597 13973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:37.771456 13973 webserver.cc:460] Webserver started at http://127.14.106.195:35399/ using document root <none> and password file <none>
I20240501 20:17:37.772276 13973 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:37.772408 13973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:37.772735 13973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:37.775549 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9d8f4452f2a249baa5abd89a81afc58a"
format_stamp: "Formatted at 2024-05-01 20:17:37 on dist-test-slave-pg4x"
I20240501 20:17:37.776333 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9d8f4452f2a249baa5abd89a81afc58a"
format_stamp: "Formatted at 2024-05-01 20:17:37 on dist-test-slave-pg4x"
I20240501 20:17:37.781733 13973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:17:37.785208 13987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:37.786670 13973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:17:37.786840 13973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9d8f4452f2a249baa5abd89a81afc58a"
format_stamp: "Formatted at 2024-05-01 20:17:37 on dist-test-slave-pg4x"
I20240501 20:17:37.787053 13973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:37.808959 13973 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:37.809835 13973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:37.810120 13973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:37.811120 13973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:37.812700 13973 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:37.812793 13973 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:37.812883 13973 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:37.812939 13973 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:37.845292 13973 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.195:36849
I20240501 20:17:37.845346 14101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.195:36849 every 8 connection(s)
I20240501 20:17:37.846750 13973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:17:37.851207 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 13973
I20240501 20:17:37.851372 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:17:37.854596 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:0
--local_ip_for_outbound_sockets=127.14.106.196
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:35537
--builtin_ntp_servers=127.14.106.212:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:17:37.859889 14102 heartbeater.cc:347] Connected to a master server at 127.14.106.254:35537
I20240501 20:17:37.860239 14102 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:37.861076 14102 heartbeater.cc:511] Master 127.14.106.254:35537 requested a full tablet report, sending...
I20240501 20:17:37.862421 13651 ts_manager.cc:194] Registered new tserver with Master: 9d8f4452f2a249baa5abd89a81afc58a (127.14.106.195:36849)
I20240501 20:17:37.863252 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.195:53769
I20240501 20:17:37.940623 13838 heartbeater.cc:503] Master 127.14.106.254:35537 was elected leader, sending a full tablet report...
W20240501 20:17:38.195240 14105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:38.195592 14105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:38.195749 14105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:38.203503 14105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:17:38.203675 14105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:38.203840 14105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:17:38.213255 14105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:35537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:38.214732 14105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:38.216574 14105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:38.228899 14110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:38.229452 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:38.229866 14111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:38.230857 14105 server_base.cc:1034] running on GCE node
I20240501 20:17:38.231609 14105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:38.232596 14105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:38.233817 14105 hybrid_clock.cc:648] HybridClock initialized: now 1714594658233770 us; error 43 us; skew 500 ppm
I20240501 20:17:38.234166 14105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:38.236572 14105 webserver.cc:460] Webserver started at http://127.14.106.196:38411/ using document root <none> and password file <none>
I20240501 20:17:38.237332 14105 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:38.237468 14105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:38.237784 14105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:38.240453 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5f0fa9ed701d4e3e94424d41e458a87f"
format_stamp: "Formatted at 2024-05-01 20:17:38 on dist-test-slave-pg4x"
I20240501 20:17:38.241204 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5f0fa9ed701d4e3e94424d41e458a87f"
format_stamp: "Formatted at 2024-05-01 20:17:38 on dist-test-slave-pg4x"
I20240501 20:17:38.246464 14105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:17:38.249917 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:38.251439 14105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:38.251616 14105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5f0fa9ed701d4e3e94424d41e458a87f"
format_stamp: "Formatted at 2024-05-01 20:17:38 on dist-test-slave-pg4x"
I20240501 20:17:38.251822 14105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:38.266314 14105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:38.267267 14105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:38.267549 14105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:38.268488 14105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:38.270000 14105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:38.270102 14105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:38.270216 14105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:38.270287 14105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:38.302026 14105 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:46597
I20240501 20:17:38.302078 14232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:46597 every 8 connection(s)
I20240501 20:17:38.303349 14105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:17:38.308125 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 14105
I20240501 20:17:38.308338 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:17:38.323940 14233 heartbeater.cc:347] Connected to a master server at 127.14.106.254:35537
I20240501 20:17:38.324267 14233 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:38.325305 14233 heartbeater.cc:511] Master 127.14.106.254:35537 requested a full tablet report, sending...
I20240501 20:17:38.327237 13650 ts_manager.cc:194] Registered new tserver with Master: 5f0fa9ed701d4e3e94424d41e458a87f (127.14.106.196:46597)
I20240501 20:17:38.328505 13650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:45787
I20240501 20:17:38.341684 14763 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:17:38.362473 13650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53842:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:17:38.401912 13970 heartbeater.cc:503] Master 127.14.106.254:35537 was elected leader, sending a full tablet report...
I20240501 20:17:38.406610 13905 tablet_service.cc:1467] Processing CreateTablet for tablet e679af9d27d24ece98b70bbb098b98b8 (DEFAULT_TABLE table=TestTable [id=a007f367c1364b15bfcc32b9944103a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:38.408370 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e679af9d27d24ece98b70bbb098b98b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:38.424978 14247 tablet_bootstrap.cc:492] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c: Bootstrap starting.
I20240501 20:17:38.427784 14247 tablet_bootstrap.cc:654] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:38.428740 14247 log.cc:824] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:17:38.431123 14247 tablet_bootstrap.cc:492] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c: No bootstrap required, opened a new log
I20240501 20:17:38.431349 14247 ts_tablet_manager.cc:1367] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:17:38.437300 14247 raft_consensus.cc:348] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:38.437608 14247 raft_consensus.cc:374] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:38.437700 14247 raft_consensus.cc:729] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Initialized, Role: FOLLOWER
I20240501 20:17:38.438230 14247 consensus_queue.cc:260] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:38.438437 14247 raft_consensus.cc:388] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:38.438557 14247 raft_consensus.cc:482] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:38.438716 14247 raft_consensus.cc:3037] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:38.440570 14247 raft_consensus.cc:504] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:38.440999 14247 leader_election.cc:302] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5726ef69254e618c4be6a1daabaf6c; no voters:
I20240501 20:17:38.441635 14247 leader_election.cc:288] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:38.441787 14249 raft_consensus.cc:2781] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:38.443727 14247 ts_tablet_manager.cc:1398] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20240501 20:17:38.443856 14249 raft_consensus.cc:686] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 LEADER]: Becoming Leader. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Running, Role: LEADER
I20240501 20:17:38.444315 14249 consensus_queue.cc:237] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:38.449476 13650 catalog_manager.cc:5484] T e679af9d27d24ece98b70bbb098b98b8 P 9e5726ef69254e618c4be6a1daabaf6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5726ef69254e618c4be6a1daabaf6c (127.14.106.194). New cstate: current_term: 1 leader_uuid: "9e5726ef69254e618c4be6a1daabaf6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } } }
I20240501 20:17:38.469864 14763 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:17:38.472265 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6588a03404be4ed19b41e92920ca66f7 to finish bootstrapping
I20240501 20:17:38.484222 14763 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9e5726ef69254e618c4be6a1daabaf6c to finish bootstrapping
I20240501 20:17:38.493935 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9d8f4452f2a249baa5abd89a81afc58a to finish bootstrapping
I20240501 20:17:38.504539 14763 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5f0fa9ed701d4e3e94424d41e458a87f to finish bootstrapping
I20240501 20:17:38.514508 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 13973
I20240501 20:17:38.515599 14097 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:38.597677 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 13973
I20240501 20:17:38.615792 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 14105
I20240501 20:17:38.616971 14228 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:38.699308 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 14105
I20240501 20:17:42.469102 13650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53842:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:17:42.487289 13771 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc44d5b089b44b8a9c36e93c0f40bd6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=02e0427cda904f81af38e4c7a795a11c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:42.487319 13773 tablet_service.cc:1467] Processing CreateTablet for tablet 9f689a4f7a914a898848d6369c7bbd18 (DEFAULT_TABLE table=rebalance_test_table_0 [id=02e0427cda904f81af38e4c7a795a11c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:42.487227 13772 tablet_service.cc:1467] Processing CreateTablet for tablet bf74b626d8614adcb3d2dfb3fa3032fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=02e0427cda904f81af38e4c7a795a11c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:42.488837 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc44d5b089b44b8a9c36e93c0f40bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:42.489286 13773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f689a4f7a914a898848d6369c7bbd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:42.489576 13772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf74b626d8614adcb3d2dfb3fa3032fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:42.500255 14266 tablet_bootstrap.cc:492] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7: Bootstrap starting.
I20240501 20:17:42.502924 14266 tablet_bootstrap.cc:654] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:42.504001 14266 log.cc:824] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:17:42.506201 14266 tablet_bootstrap.cc:492] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7: No bootstrap required, opened a new log
I20240501 20:17:42.506444 14266 ts_tablet_manager.cc:1367] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:17:42.512452 14266 raft_consensus.cc:348] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.512738 14266 raft_consensus.cc:374] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:42.512837 14266 raft_consensus.cc:729] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Initialized, Role: FOLLOWER
I20240501 20:17:42.513417 14266 consensus_queue.cc:260] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.513617 14266 raft_consensus.cc:388] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:42.513743 14266 raft_consensus.cc:482] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:42.513926 14266 raft_consensus.cc:3037] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:42.515866 14266 raft_consensus.cc:504] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.516302 14266 leader_election.cc:302] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6588a03404be4ed19b41e92920ca66f7; no voters:
I20240501 20:17:42.516827 14266 leader_election.cc:288] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:42.517027 14268 raft_consensus.cc:2781] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:42.517642 14268 raft_consensus.cc:686] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Running, Role: LEADER
I20240501 20:17:42.518148 14268 consensus_queue.cc:237] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.518885 14266 ts_tablet_manager.cc:1398] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20240501 20:17:42.519246 14266 tablet_bootstrap.cc:492] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7: Bootstrap starting.
I20240501 20:17:42.521250 14266 tablet_bootstrap.cc:654] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:42.522534 13650 catalog_manager.cc:5484] T 9fc44d5b089b44b8a9c36e93c0f40bd6 P 6588a03404be4ed19b41e92920ca66f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6588a03404be4ed19b41e92920ca66f7 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "6588a03404be4ed19b41e92920ca66f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } } }
I20240501 20:17:42.523411 14266 tablet_bootstrap.cc:492] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7: No bootstrap required, opened a new log
I20240501 20:17:42.523556 14266 ts_tablet_manager.cc:1367] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:17:42.524376 14266 raft_consensus.cc:348] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.524513 14266 raft_consensus.cc:374] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:42.524602 14266 raft_consensus.cc:729] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Initialized, Role: FOLLOWER
I20240501 20:17:42.524811 14266 consensus_queue.cc:260] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.524979 14266 raft_consensus.cc:388] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:42.525054 14266 raft_consensus.cc:482] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:42.525163 14266 raft_consensus.cc:3037] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:42.527163 14266 raft_consensus.cc:504] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.527375 14266 leader_election.cc:302] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6588a03404be4ed19b41e92920ca66f7; no voters:
I20240501 20:17:42.527581 14266 leader_election.cc:288] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:42.527628 14268 raft_consensus.cc:2781] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:42.527808 14268 raft_consensus.cc:686] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Running, Role: LEADER
I20240501 20:17:42.528033 14266 ts_tablet_manager.cc:1398] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:17:42.528039 14268 consensus_queue.cc:237] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.528311 14266 tablet_bootstrap.cc:492] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7: Bootstrap starting.
I20240501 20:17:42.530110 14266 tablet_bootstrap.cc:654] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:42.530320 13651 catalog_manager.cc:5484] T bf74b626d8614adcb3d2dfb3fa3032fc P 6588a03404be4ed19b41e92920ca66f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6588a03404be4ed19b41e92920ca66f7 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "6588a03404be4ed19b41e92920ca66f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } } }
I20240501 20:17:42.531939 14266 tablet_bootstrap.cc:492] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7: No bootstrap required, opened a new log
I20240501 20:17:42.532101 14266 ts_tablet_manager.cc:1367] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:17:42.532629 14266 raft_consensus.cc:348] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.532801 14266 raft_consensus.cc:374] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:42.532887 14266 raft_consensus.cc:729] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Initialized, Role: FOLLOWER
I20240501 20:17:42.533115 14266 consensus_queue.cc:260] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.533269 14266 raft_consensus.cc:388] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:42.533396 14266 raft_consensus.cc:482] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:42.533511 14266 raft_consensus.cc:3037] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:42.535336 14266 raft_consensus.cc:504] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.535530 14266 leader_election.cc:302] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6588a03404be4ed19b41e92920ca66f7; no voters:
I20240501 20:17:42.535686 14266 leader_election.cc:288] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:42.535766 14268 raft_consensus.cc:2781] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:42.535940 14268 raft_consensus.cc:686] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Running, Role: LEADER
I20240501 20:17:42.536120 14266 ts_tablet_manager.cc:1398] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:17:42.536178 14268 consensus_queue.cc:237] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:42.538038 13651 catalog_manager.cc:5484] T 9f689a4f7a914a898848d6369c7bbd18 P 6588a03404be4ed19b41e92920ca66f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6588a03404be4ed19b41e92920ca66f7 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "6588a03404be4ed19b41e92920ca66f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } } }
I20240501 20:17:43.977234 13651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53842:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:17:43.986671 13905 tablet_service.cc:1467] Processing CreateTablet for tablet 86ab217689b74f1aa36e3de4a9d9e65f (DEFAULT_TABLE table=rebalance_test_table_1 [id=54cc07242eda44f4b4564c1443301e3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:43.986783 13904 tablet_service.cc:1467] Processing CreateTablet for tablet f459bfffa37f4e1e98c62182d48ac3da (DEFAULT_TABLE table=rebalance_test_table_1 [id=54cc07242eda44f4b4564c1443301e3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:43.987135 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ab217689b74f1aa36e3de4a9d9e65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:43.987210 13903 tablet_service.cc:1467] Processing CreateTablet for tablet 53b9631c70434449806cb18d0450f155 (DEFAULT_TABLE table=rebalance_test_table_1 [id=54cc07242eda44f4b4564c1443301e3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:43.987375 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f459bfffa37f4e1e98c62182d48ac3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:43.987718 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b9631c70434449806cb18d0450f155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:43.993605 14302 tablet_bootstrap.cc:492] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c: Bootstrap starting.
I20240501 20:17:43.995249 14302 tablet_bootstrap.cc:654] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:43.996876 14302 tablet_bootstrap.cc:492] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c: No bootstrap required, opened a new log
I20240501 20:17:43.996989 14302 ts_tablet_manager.cc:1367] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:43.997697 14302 raft_consensus.cc:348] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:43.997828 14302 raft_consensus.cc:374] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:43.997924 14302 raft_consensus.cc:729] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Initialized, Role: FOLLOWER
I20240501 20:17:43.998119 14302 consensus_queue.cc:260] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:43.998252 14302 raft_consensus.cc:388] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:43.998320 14302 raft_consensus.cc:482] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:43.998416 14302 raft_consensus.cc:3037] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:44.000278 14302 raft_consensus.cc:504] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.000497 14302 leader_election.cc:302] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5726ef69254e618c4be6a1daabaf6c; no voters:
I20240501 20:17:44.000873 14302 leader_election.cc:288] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:44.001024 14304 raft_consensus.cc:2781] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:44.001572 14304 raft_consensus.cc:686] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 LEADER]: Becoming Leader. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Running, Role: LEADER
I20240501 20:17:44.001848 14302 ts_tablet_manager.cc:1398] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240501 20:17:44.001914 14304 consensus_queue.cc:237] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.002079 14302 tablet_bootstrap.cc:492] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c: Bootstrap starting.
I20240501 20:17:44.003922 14302 tablet_bootstrap.cc:654] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:44.004266 13651 catalog_manager.cc:5484] T 53b9631c70434449806cb18d0450f155 P 9e5726ef69254e618c4be6a1daabaf6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5726ef69254e618c4be6a1daabaf6c (127.14.106.194). New cstate: current_term: 1 leader_uuid: "9e5726ef69254e618c4be6a1daabaf6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } } }
I20240501 20:17:44.005753 14302 tablet_bootstrap.cc:492] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c: No bootstrap required, opened a new log
I20240501 20:17:44.005898 14302 ts_tablet_manager.cc:1367] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:17:44.006533 14302 raft_consensus.cc:348] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.006706 14302 raft_consensus.cc:374] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:44.006786 14302 raft_consensus.cc:729] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Initialized, Role: FOLLOWER
I20240501 20:17:44.007002 14302 consensus_queue.cc:260] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.007128 14302 raft_consensus.cc:388] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:44.007241 14302 raft_consensus.cc:482] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:44.007334 14302 raft_consensus.cc:3037] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:44.009234 14302 raft_consensus.cc:504] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.009415 14302 leader_election.cc:302] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5726ef69254e618c4be6a1daabaf6c; no voters:
I20240501 20:17:44.009599 14302 leader_election.cc:288] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:44.009656 14304 raft_consensus.cc:2781] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:44.009843 14304 raft_consensus.cc:686] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [term 1 LEADER]: Becoming Leader. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Running, Role: LEADER
I20240501 20:17:44.010061 14302 ts_tablet_manager.cc:1398] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:17:44.010102 14304 consensus_queue.cc:237] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.010495 14302 tablet_bootstrap.cc:492] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c: Bootstrap starting.
I20240501 20:17:44.012130 14302 tablet_bootstrap.cc:654] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:44.012311 13651 catalog_manager.cc:5484] T 86ab217689b74f1aa36e3de4a9d9e65f P 9e5726ef69254e618c4be6a1daabaf6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5726ef69254e618c4be6a1daabaf6c (127.14.106.194). New cstate: current_term: 1 leader_uuid: "9e5726ef69254e618c4be6a1daabaf6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } } }
I20240501 20:17:44.013888 14302 tablet_bootstrap.cc:492] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c: No bootstrap required, opened a new log
I20240501 20:17:44.014020 14302 ts_tablet_manager.cc:1367] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:17:44.014786 14302 raft_consensus.cc:348] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.014938 14302 raft_consensus.cc:374] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:44.015030 14302 raft_consensus.cc:729] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Initialized, Role: FOLLOWER
I20240501 20:17:44.015281 14302 consensus_queue.cc:260] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.015488 14302 raft_consensus.cc:388] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:44.015573 14302 raft_consensus.cc:482] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:44.015681 14302 raft_consensus.cc:3037] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:44.017558 14302 raft_consensus.cc:504] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.017771 14302 leader_election.cc:302] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5726ef69254e618c4be6a1daabaf6c; no voters:
I20240501 20:17:44.017942 14302 leader_election.cc:288] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:44.018043 14304 raft_consensus.cc:2781] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:44.018198 14304 raft_consensus.cc:686] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [term 1 LEADER]: Becoming Leader. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Running, Role: LEADER
I20240501 20:17:44.018419 14304 consensus_queue.cc:237] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:44.018541 14302 ts_tablet_manager.cc:1398] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240501 20:17:44.020145 13651 catalog_manager.cc:5484] T f459bfffa37f4e1e98c62182d48ac3da P 9e5726ef69254e618c4be6a1daabaf6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5726ef69254e618c4be6a1daabaf6c (127.14.106.194). New cstate: current_term: 1 leader_uuid: "9e5726ef69254e618c4be6a1daabaf6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } } }
I20240501 20:17:44.526661 13651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53842:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:17:44.534934 13771 tablet_service.cc:1467] Processing CreateTablet for tablet 1927c534a1f54233bc76a9416454d080 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1c48f74992cc451fb7a2c8819bb77874]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:44.534890 13772 tablet_service.cc:1467] Processing CreateTablet for tablet 09589981fafe468ab97ae7939af21f72 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1c48f74992cc451fb7a2c8819bb77874]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:44.535444 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1927c534a1f54233bc76a9416454d080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:44.535703 13772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09589981fafe468ab97ae7939af21f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:44.535696 13773 tablet_service.cc:1467] Processing CreateTablet for tablet d8cd0c5231ab407088b3e1e64fe1cca4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1c48f74992cc451fb7a2c8819bb77874]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:44.536011 13773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8cd0c5231ab407088b3e1e64fe1cca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:44.542294 14337 tablet_bootstrap.cc:492] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7: Bootstrap starting.
I20240501 20:17:44.543603 14337 tablet_bootstrap.cc:654] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:44.545254 14337 tablet_bootstrap.cc:492] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7: No bootstrap required, opened a new log
I20240501 20:17:44.545389 14337 ts_tablet_manager.cc:1367] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:17:44.545969 14337 raft_consensus.cc:348] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.546106 14337 raft_consensus.cc:374] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:44.546204 14337 raft_consensus.cc:729] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Initialized, Role: FOLLOWER
I20240501 20:17:44.546427 14337 consensus_queue.cc:260] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.546576 14337 raft_consensus.cc:388] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:44.546689 14337 raft_consensus.cc:482] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:44.546793 14337 raft_consensus.cc:3037] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:44.548651 14337 raft_consensus.cc:504] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.548887 14337 leader_election.cc:302] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6588a03404be4ed19b41e92920ca66f7; no voters:
I20240501 20:17:44.549248 14337 leader_election.cc:288] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:44.549448 14339 raft_consensus.cc:2781] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:44.549867 14337 ts_tablet_manager.cc:1398] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240501 20:17:44.550051 14339 raft_consensus.cc:686] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Running, Role: LEADER
I20240501 20:17:44.550113 14337 tablet_bootstrap.cc:492] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7: Bootstrap starting.
I20240501 20:17:44.550333 14339 consensus_queue.cc:237] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.551676 14337 tablet_bootstrap.cc:654] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:44.552209 13651 catalog_manager.cc:5484] T d8cd0c5231ab407088b3e1e64fe1cca4 P 6588a03404be4ed19b41e92920ca66f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6588a03404be4ed19b41e92920ca66f7 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "6588a03404be4ed19b41e92920ca66f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } } }
I20240501 20:17:44.553534 14337 tablet_bootstrap.cc:492] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7: No bootstrap required, opened a new log
I20240501 20:17:44.553674 14337 ts_tablet_manager.cc:1367] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:17:44.554194 14337 raft_consensus.cc:348] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.554339 14337 raft_consensus.cc:374] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:44.554421 14337 raft_consensus.cc:729] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Initialized, Role: FOLLOWER
I20240501 20:17:44.554674 14337 consensus_queue.cc:260] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.554826 14337 raft_consensus.cc:388] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:44.554936 14337 raft_consensus.cc:482] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:44.555040 14337 raft_consensus.cc:3037] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:44.557425 14337 raft_consensus.cc:504] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.557614 14337 leader_election.cc:302] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6588a03404be4ed19b41e92920ca66f7; no voters:
I20240501 20:17:44.557796 14337 leader_election.cc:288] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:44.557830 14339 raft_consensus.cc:2781] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:44.558043 14339 raft_consensus.cc:686] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Running, Role: LEADER
I20240501 20:17:44.558272 14337 ts_tablet_manager.cc:1398] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240501 20:17:44.558274 14339 consensus_queue.cc:237] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.558542 14337 tablet_bootstrap.cc:492] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7: Bootstrap starting.
I20240501 20:17:44.560038 14337 tablet_bootstrap.cc:654] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:44.560556 13651 catalog_manager.cc:5484] T 09589981fafe468ab97ae7939af21f72 P 6588a03404be4ed19b41e92920ca66f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6588a03404be4ed19b41e92920ca66f7 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "6588a03404be4ed19b41e92920ca66f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } } }
I20240501 20:17:44.561664 14337 tablet_bootstrap.cc:492] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7: No bootstrap required, opened a new log
I20240501 20:17:44.561815 14337 ts_tablet_manager.cc:1367] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240501 20:17:44.562443 14337 raft_consensus.cc:348] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.562578 14337 raft_consensus.cc:374] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:44.562685 14337 raft_consensus.cc:729] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Initialized, Role: FOLLOWER
I20240501 20:17:44.562943 14337 consensus_queue.cc:260] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.563148 14337 raft_consensus.cc:388] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:44.563236 14337 raft_consensus.cc:482] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:44.563345 14337 raft_consensus.cc:3037] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:44.565583 14337 raft_consensus.cc:504] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.565768 14337 leader_election.cc:302] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6588a03404be4ed19b41e92920ca66f7; no voters:
I20240501 20:17:44.565913 14337 leader_election.cc:288] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:44.566015 14339 raft_consensus.cc:2781] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:44.566234 14339 raft_consensus.cc:686] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6588a03404be4ed19b41e92920ca66f7, State: Running, Role: LEADER
I20240501 20:17:44.566318 14337 ts_tablet_manager.cc:1398] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:17:44.566476 14339 consensus_queue.cc:237] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } }
I20240501 20:17:44.568234 13651 catalog_manager.cc:5484] T 1927c534a1f54233bc76a9416454d080 P 6588a03404be4ed19b41e92920ca66f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6588a03404be4ed19b41e92920ca66f7 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "6588a03404be4ed19b41e92920ca66f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6588a03404be4ed19b41e92920ca66f7" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 34519 } } }
I20240501 20:17:45.086727 13650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53842:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:17:45.095082 13904 tablet_service.cc:1467] Processing CreateTablet for tablet 2869a280b72042e19ee1b4f72fa966ca (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e2520747194dc2a075839fe2d4ebc8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:45.095242 13905 tablet_service.cc:1467] Processing CreateTablet for tablet 12df2c92830d4ba28c355f8121168179 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e2520747194dc2a075839fe2d4ebc8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:45.095343 13903 tablet_service.cc:1467] Processing CreateTablet for tablet 4fee0cb3e0a544d0862b082ed522f947 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e2520747194dc2a075839fe2d4ebc8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:45.095571 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2869a280b72042e19ee1b4f72fa966ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:45.095784 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12df2c92830d4ba28c355f8121168179. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:45.096062 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fee0cb3e0a544d0862b082ed522f947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:45.103282 14373 tablet_bootstrap.cc:492] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c: Bootstrap starting.
I20240501 20:17:45.104539 14373 tablet_bootstrap.cc:654] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:45.106076 14373 tablet_bootstrap.cc:492] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c: No bootstrap required, opened a new log
I20240501 20:17:45.106195 14373 ts_tablet_manager.cc:1367] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:17:45.106945 14373 raft_consensus.cc:348] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.107084 14373 raft_consensus.cc:374] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:45.107175 14373 raft_consensus.cc:729] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Initialized, Role: FOLLOWER
I20240501 20:17:45.107393 14373 consensus_queue.cc:260] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.107537 14373 raft_consensus.cc:388] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:45.107612 14373 raft_consensus.cc:482] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:45.107715 14373 raft_consensus.cc:3037] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:45.109434 14373 raft_consensus.cc:504] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.109661 14373 leader_election.cc:302] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5726ef69254e618c4be6a1daabaf6c; no voters:
I20240501 20:17:45.110092 14373 leader_election.cc:288] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:45.110321 14375 raft_consensus.cc:2781] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:45.110806 14373 ts_tablet_manager.cc:1398] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 20:17:45.110977 14375 raft_consensus.cc:686] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [term 1 LEADER]: Becoming Leader. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Running, Role: LEADER
I20240501 20:17:45.111079 14373 tablet_bootstrap.cc:492] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c: Bootstrap starting.
I20240501 20:17:45.111246 14375 consensus_queue.cc:237] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.112748 14373 tablet_bootstrap.cc:654] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:45.113510 13650 catalog_manager.cc:5484] T 2869a280b72042e19ee1b4f72fa966ca P 9e5726ef69254e618c4be6a1daabaf6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5726ef69254e618c4be6a1daabaf6c (127.14.106.194). New cstate: current_term: 1 leader_uuid: "9e5726ef69254e618c4be6a1daabaf6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } } }
I20240501 20:17:45.115155 14373 tablet_bootstrap.cc:492] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c: No bootstrap required, opened a new log
I20240501 20:17:45.115326 14373 ts_tablet_manager.cc:1367] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:17:45.115921 14373 raft_consensus.cc:348] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.116102 14373 raft_consensus.cc:374] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:45.116184 14373 raft_consensus.cc:729] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Initialized, Role: FOLLOWER
I20240501 20:17:45.116380 14373 consensus_queue.cc:260] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.116504 14373 raft_consensus.cc:388] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:45.116567 14373 raft_consensus.cc:482] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:45.116649 14373 raft_consensus.cc:3037] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:45.118826 14373 raft_consensus.cc:504] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.119099 14373 leader_election.cc:302] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5726ef69254e618c4be6a1daabaf6c; no voters:
I20240501 20:17:45.119324 14373 leader_election.cc:288] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:45.119601 14377 raft_consensus.cc:2781] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:45.119793 14373 ts_tablet_manager.cc:1398] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240501 20:17:45.119800 14377 raft_consensus.cc:686] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 LEADER]: Becoming Leader. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Running, Role: LEADER
I20240501 20:17:45.120106 14373 tablet_bootstrap.cc:492] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c: Bootstrap starting.
I20240501 20:17:45.120246 14377 consensus_queue.cc:237] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.121698 14373 tablet_bootstrap.cc:654] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:45.122793 13650 catalog_manager.cc:5484] T 4fee0cb3e0a544d0862b082ed522f947 P 9e5726ef69254e618c4be6a1daabaf6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5726ef69254e618c4be6a1daabaf6c (127.14.106.194). New cstate: current_term: 1 leader_uuid: "9e5726ef69254e618c4be6a1daabaf6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } } }
I20240501 20:17:45.123816 14373 tablet_bootstrap.cc:492] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c: No bootstrap required, opened a new log
I20240501 20:17:45.123939 14373 ts_tablet_manager.cc:1367] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:17:45.124513 14373 raft_consensus.cc:348] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.124670 14373 raft_consensus.cc:374] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:45.124766 14373 raft_consensus.cc:729] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Initialized, Role: FOLLOWER
I20240501 20:17:45.124992 14373 consensus_queue.cc:260] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.125187 14373 raft_consensus.cc:388] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:45.125267 14373 raft_consensus.cc:482] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:45.125375 14373 raft_consensus.cc:3037] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:45.127826 14373 raft_consensus.cc:504] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.128059 14373 leader_election.cc:302] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5726ef69254e618c4be6a1daabaf6c; no voters:
I20240501 20:17:45.128234 14373 leader_election.cc:288] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:45.128294 14377 raft_consensus.cc:2781] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:45.128516 14377 raft_consensus.cc:686] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [term 1 LEADER]: Becoming Leader. State: Replica: 9e5726ef69254e618c4be6a1daabaf6c, State: Running, Role: LEADER
I20240501 20:17:45.128687 14373 ts_tablet_manager.cc:1398] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:17:45.128767 14377 consensus_queue.cc:237] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } }
I20240501 20:17:45.130841 13650 catalog_manager.cc:5484] T 12df2c92830d4ba28c355f8121168179 P 9e5726ef69254e618c4be6a1daabaf6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5726ef69254e618c4be6a1daabaf6c (127.14.106.194). New cstate: current_term: 1 leader_uuid: "9e5726ef69254e618c4be6a1daabaf6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5726ef69254e618c4be6a1daabaf6c" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 38669 } } }
I20240501 20:17:45.633234 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.195:36849
--local_ip_for_outbound_sockets=127.14.106.195
--tserver_master_addrs=127.14.106.254:35537
--webserver_port=35399
--webserver_interface=127.14.106.195
--builtin_ntp_servers=127.14.106.212:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:17:45.973309 14406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:45.973616 14406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:45.973762 14406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:45.981362 14406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:17:45.981494 14406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:45.981638 14406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.195
I20240501 20:17:45.990669 14406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.195:36849
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.106.195
--webserver_port=35399
--tserver_master_addrs=127.14.106.254:35537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.195
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:45.991976 14406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:45.993667 14406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:46.005692 14411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:46.006268 14412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:46.006948 14414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:46.007623 14406 server_base.cc:1034] running on GCE node
I20240501 20:17:46.008247 14406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:46.009155 14406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:46.010355 14406 hybrid_clock.cc:648] HybridClock initialized: now 1714594666010312 us; error 37 us; skew 500 ppm
I20240501 20:17:46.010704 14406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:46.012861 14406 webserver.cc:460] Webserver started at http://127.14.106.195:35399/ using document root <none> and password file <none>
I20240501 20:17:46.013597 14406 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:46.013731 14406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:46.018893 14406 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:46.021534 14420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:46.022884 14406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:46.023041 14406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9d8f4452f2a249baa5abd89a81afc58a"
format_stamp: "Formatted at 2024-05-01 20:17:37 on dist-test-slave-pg4x"
I20240501 20:17:46.023968 14406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:46.039844 14406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:46.040673 14406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:46.040953 14406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:46.041908 14406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:46.043491 14406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:46.043571 14406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:46.043704 14406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:46.043766 14406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:46.076344 14406 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.195:36849
I20240501 20:17:46.076424 14532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.195:36849 every 8 connection(s)
I20240501 20:17:46.077888 14406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:17:46.085232 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 14406
I20240501 20:17:46.086055 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:46597
--local_ip_for_outbound_sockets=127.14.106.196
--tserver_master_addrs=127.14.106.254:35537
--webserver_port=38411
--webserver_interface=127.14.106.196
--builtin_ntp_servers=127.14.106.212:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:17:46.091014 14533 heartbeater.cc:347] Connected to a master server at 127.14.106.254:35537
I20240501 20:17:46.091326 14533 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:46.092085 14533 heartbeater.cc:511] Master 127.14.106.254:35537 requested a full tablet report, sending...
I20240501 20:17:46.093654 13650 ts_manager.cc:194] Re-registered known tserver with Master: 9d8f4452f2a249baa5abd89a81afc58a (127.14.106.195:36849)
I20240501 20:17:46.094470 13650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.195:46309
W20240501 20:17:46.441619 14537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:46.441895 14537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:46.442040 14537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:46.449458 14537 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:17:46.449581 14537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:46.449756 14537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:17:46.458549 14537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:46597
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=38411
--tserver_master_addrs=127.14.106.254:35537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:46.459884 14537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:46.461640 14537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:46.473484 14542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:46.474114 14545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:46.474838 14543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:46.475502 14537 server_base.cc:1034] running on GCE node
I20240501 20:17:46.476060 14537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:46.477017 14537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:46.478227 14537 hybrid_clock.cc:648] HybridClock initialized: now 1714594666478161 us; error 63 us; skew 500 ppm
I20240501 20:17:46.478536 14537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:46.480688 14537 webserver.cc:460] Webserver started at http://127.14.106.196:38411/ using document root <none> and password file <none>
I20240501 20:17:46.481389 14537 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:46.481526 14537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:46.486480 14537 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:17:46.489106 14551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:46.490382 14537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:17:46.490538 14537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5f0fa9ed701d4e3e94424d41e458a87f"
format_stamp: "Formatted at 2024-05-01 20:17:38 on dist-test-slave-pg4x"
I20240501 20:17:46.491369 14537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:46.519919 14537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:46.520732 14537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:46.521015 14537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:46.521941 14537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:46.523464 14537 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:46.523545 14537 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:46.523676 14537 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:46.523734 14537 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:46.555645 14537 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:46597
I20240501 20:17:46.555703 14663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:46597 every 8 connection(s)
I20240501 20:17:46.557003 14537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:17:46.559820 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 14537
I20240501 20:17:46.570187 14664 heartbeater.cc:347] Connected to a master server at 127.14.106.254:35537
I20240501 20:17:46.570513 14664 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:46.571234 14664 heartbeater.cc:511] Master 127.14.106.254:35537 requested a full tablet report, sending...
I20240501 20:17:46.572619 13650 ts_manager.cc:194] Re-registered known tserver with Master: 5f0fa9ed701d4e3e94424d41e458a87f (127.14.106.196:46597)
I20240501 20:17:46.573747 13650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:47769
I20240501 20:17:47.061113 13771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:47.062198 14468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:47.064499 14599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:47.065284 13904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:47.096879 14533 heartbeater.cc:503] Master 127.14.106.254:35537 was elected leader, sending a full tablet report...
I20240501 20:17:47.238143 14599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:47.242727 13904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:47.243156 13771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:47.243788 14468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:47.424748 14599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:47.425312 13771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:47.430693 13904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:47.440732 14468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:47.576570 14664 heartbeater.cc:503] Master 127.14.106.254:35537 was elected leader, sending a full tablet report...
I20240501 20:17:47.716336 14599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:17:47.720157 13771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:17:47.722947 13904 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:17:47.736476 14468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
aa0a736e8ebf4701a363da1d56b165b7 | 127.14.106.254:35537 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.106.212:41181 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
5f0fa9ed701d4e3e94424d41e458a87f | 127.14.106.196:46597 | HEALTHY | <none> | 0 | 0
6588a03404be4ed19b41e92920ca66f7 | 127.14.106.193:34519 | HEALTHY | <none> | 6 | 0
9d8f4452f2a249baa5abd89a81afc58a | 127.14.106.195:36849 | HEALTHY | <none> | 0 | 0
9e5726ef69254e618c4be6a1daabaf6c | 127.14.106.194:38669 | HEALTHY | <none> | 7 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.14.106.193 | experimental | 127.14.106.193:34519
local_ip_for_outbound_sockets | 127.14.106.194 | experimental | 127.14.106.194:38669
local_ip_for_outbound_sockets | 127.14.106.195 | experimental | 127.14.106.195:36849
local_ip_for_outbound_sockets | 127.14.106.196 | experimental | 127.14.106.196:46597
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.106.193:34519
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.106.194:38669
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.106.195:36849
server_dump_info_path | /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.106.196:46597
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.14.106.212:41181 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:17:47.780222 14763 log_verifier.cc:126] Checking tablet 09589981fafe468ab97ae7939af21f72
I20240501 20:17:47.788393 14763 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 09589981fafe468ab97ae7939af21f72
I20240501 20:17:47.788488 14763 log_verifier.cc:126] Checking tablet 12df2c92830d4ba28c355f8121168179
I20240501 20:17:47.796164 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 12df2c92830d4ba28c355f8121168179
I20240501 20:17:47.796257 14763 log_verifier.cc:126] Checking tablet 1927c534a1f54233bc76a9416454d080
I20240501 20:17:47.845880 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 1927c534a1f54233bc76a9416454d080
I20240501 20:17:47.845968 14763 log_verifier.cc:126] Checking tablet 2869a280b72042e19ee1b4f72fa966ca
I20240501 20:17:47.854128 14763 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2869a280b72042e19ee1b4f72fa966ca
I20240501 20:17:47.854238 14763 log_verifier.cc:126] Checking tablet 4fee0cb3e0a544d0862b082ed522f947
I20240501 20:17:47.861714 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 4fee0cb3e0a544d0862b082ed522f947
I20240501 20:17:47.861804 14763 log_verifier.cc:126] Checking tablet 53b9631c70434449806cb18d0450f155
I20240501 20:17:47.869393 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 53b9631c70434449806cb18d0450f155
I20240501 20:17:47.869479 14763 log_verifier.cc:126] Checking tablet 86ab217689b74f1aa36e3de4a9d9e65f
I20240501 20:17:47.877221 14763 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 86ab217689b74f1aa36e3de4a9d9e65f
I20240501 20:17:47.877306 14763 log_verifier.cc:126] Checking tablet 9f689a4f7a914a898848d6369c7bbd18
I20240501 20:17:47.885015 14763 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 9f689a4f7a914a898848d6369c7bbd18
I20240501 20:17:47.885107 14763 log_verifier.cc:126] Checking tablet 9fc44d5b089b44b8a9c36e93c0f40bd6
I20240501 20:17:47.892729 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9fc44d5b089b44b8a9c36e93c0f40bd6
I20240501 20:17:47.892822 14763 log_verifier.cc:126] Checking tablet bf74b626d8614adcb3d2dfb3fa3032fc
I20240501 20:17:47.900436 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet bf74b626d8614adcb3d2dfb3fa3032fc
I20240501 20:17:47.900523 14763 log_verifier.cc:126] Checking tablet d8cd0c5231ab407088b3e1e64fe1cca4
I20240501 20:17:47.908470 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d8cd0c5231ab407088b3e1e64fe1cca4
I20240501 20:17:47.908557 14763 log_verifier.cc:126] Checking tablet e679af9d27d24ece98b70bbb098b98b8
I20240501 20:17:47.916888 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e679af9d27d24ece98b70bbb098b98b8
I20240501 20:17:47.917004 14763 log_verifier.cc:126] Checking tablet f459bfffa37f4e1e98c62182d48ac3da
I20240501 20:17:47.927599 14763 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f459bfffa37f4e1e98c62182d48ac3da
I20240501 20:17:47.928598 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 13710
I20240501 20:17:47.929958 13833 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:48.034556 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 13710
I20240501 20:17:48.055133 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 13842
I20240501 20:17:48.056188 13964 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:48.158957 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 13842
I20240501 20:17:48.179338 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 14406
I20240501 20:17:48.189731 14527 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:48.278052 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 14406
I20240501 20:17:48.296950 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 14537
I20240501 20:17:48.302881 14658 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:48.395948 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 14537
I20240501 20:17:48.414928 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 13620
I20240501 20:17:48.416074 13681 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:48.535523 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 13620
2024-05-01T20:17:48Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (12779 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (12779 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240501 20:17:48.593408 14763 test_util.cc:268] Using random seed: 774357226
I20240501 20:17:48.595369 14763 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:17:48.595499 14763 ts_itest-base.cc:113] --------------
I20240501 20:17:48.595589 14763 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:17:48.595639 14763 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:17:48.595705 14763 ts_itest-base.cc:116] --------------
2024-05-01T20:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:17:48Z Disabled control of system clock
I20240501 20:17:48.653141 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:39919
--webserver_interface=127.14.106.254
--webserver_port=0
--builtin_ntp_servers=127.14.106.212:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.106.254:39919
--location_mapping_cmd=/tmp/dist-test-task8oH_XT/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 20:17:48.994760 14824 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:17:48.995074 14824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:48.995148 14824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:48.995221 14824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:49.002681 14824 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:17:49.002794 14824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:17:49.002867 14824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:49.002915 14824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:17:49.002975 14824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:17:49.011974 14824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task8oH_XT/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.14.106.254:39919
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:39919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:49.013239 14824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:49.015072 14824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:49.026223 14829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:49.026772 14830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:49.027390 14832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:49.028084 14824 server_base.cc:1034] running on GCE node
I20240501 20:17:49.028779 14824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:49.030093 14824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:49.031311 14824 hybrid_clock.cc:648] HybridClock initialized: now 1714594669031236 us; error 43 us; skew 500 ppm
I20240501 20:17:49.031647 14824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:49.033953 14824 webserver.cc:460] Webserver started at http://127.14.106.254:39249/ using document root <none> and password file <none>
I20240501 20:17:49.034669 14824 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:49.034804 14824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:49.035101 14824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:49.037634 14824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6320db961c4f4cc8925ef8d41ad87036"
format_stamp: "Formatted at 2024-05-01 20:17:49 on dist-test-slave-pg4x"
I20240501 20:17:49.038401 14824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6320db961c4f4cc8925ef8d41ad87036"
format_stamp: "Formatted at 2024-05-01 20:17:49 on dist-test-slave-pg4x"
I20240501 20:17:49.043370 14824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240501 20:17:49.046546 14838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:49.047852 14824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:17:49.048027 14824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6320db961c4f4cc8925ef8d41ad87036"
format_stamp: "Formatted at 2024-05-01 20:17:49 on dist-test-slave-pg4x"
I20240501 20:17:49.048214 14824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:49.068558 14824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:49.069345 14824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:49.069629 14824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:49.086689 14824 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.254:39919
I20240501 20:17:49.086738 14889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.254:39919 every 8 connection(s)
I20240501 20:17:49.088022 14824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:17:49.091502 14890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:49.095644 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 14824
I20240501 20:17:49.095860 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:17:49.101050 14890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036: Bootstrap starting.
I20240501 20:17:49.104413 14890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:49.105717 14890 log.cc:824] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036: Log is configured to *not* fsync() on all Append() calls
I20240501 20:17:49.108276 14890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036: No bootstrap required, opened a new log
I20240501 20:17:49.114288 14890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6320db961c4f4cc8925ef8d41ad87036" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 39919 } }
I20240501 20:17:49.114647 14890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:49.114759 14890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6320db961c4f4cc8925ef8d41ad87036, State: Initialized, Role: FOLLOWER
I20240501 20:17:49.115383 14890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6320db961c4f4cc8925ef8d41ad87036" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 39919 } }
I20240501 20:17:49.115639 14890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:17:49.115744 14890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:17:49.115880 14890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:49.117802 14890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6320db961c4f4cc8925ef8d41ad87036" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 39919 } }
I20240501 20:17:49.118224 14890 leader_election.cc:302] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6320db961c4f4cc8925ef8d41ad87036; no voters:
I20240501 20:17:49.118819 14890 leader_election.cc:288] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:17:49.118969 14895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:49.119602 14895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [term 1 LEADER]: Becoming Leader. State: Replica: 6320db961c4f4cc8925ef8d41ad87036, State: Running, Role: LEADER
I20240501 20:17:49.120081 14895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6320db961c4f4cc8925ef8d41ad87036" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 39919 } }
I20240501 20:17:49.120644 14890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:17:49.122754 14897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6320db961c4f4cc8925ef8d41ad87036. Latest consensus state: current_term: 1 leader_uuid: "6320db961c4f4cc8925ef8d41ad87036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6320db961c4f4cc8925ef8d41ad87036" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 39919 } } }
I20240501 20:17:49.123329 14897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [sys.catalog]: This master's current role is: LEADER
I20240501 20:17:49.123245 14896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6320db961c4f4cc8925ef8d41ad87036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6320db961c4f4cc8925ef8d41ad87036" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 39919 } } }
I20240501 20:17:49.123544 14896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036 [sys.catalog]: This master's current role is: LEADER
I20240501 20:17:49.131275 14902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:17:49.136433 14902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:17:49.144215 14902 catalog_manager.cc:1347] Generated new cluster ID: 7ed1f9b27fd245c7a6562e4275fc54d1
I20240501 20:17:49.144340 14902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:17:49.171573 14902 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:17:49.173305 14902 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:17:49.184458 14902 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6320db961c4f4cc8925ef8d41ad87036: Generated new TSK 0
I20240501 20:17:49.185520 14902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:17:49.191610 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.193:0
--local_ip_for_outbound_sockets=127.14.106.193
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--builtin_ntp_servers=127.14.106.212:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:17:49.540342 14914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:49.540663 14914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:49.540807 14914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:49.548527 14914 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:17:49.548686 14914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:49.548846 14914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.193
I20240501 20:17:49.558183 14914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.193
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:49.559612 14914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:49.561370 14914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:49.573918 14919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:49.574401 14920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:49.575080 14922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:49.575817 14914 server_base.cc:1034] running on GCE node
I20240501 20:17:49.576407 14914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:49.577327 14914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:49.578541 14914 hybrid_clock.cc:648] HybridClock initialized: now 1714594669578497 us; error 34 us; skew 500 ppm
I20240501 20:17:49.578915 14914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:49.581229 14914 webserver.cc:460] Webserver started at http://127.14.106.193:40087/ using document root <none> and password file <none>
I20240501 20:17:49.582119 14914 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:49.582242 14914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:49.582584 14914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:49.585229 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ee984883adca4b1fada1f685b5275ca0"
format_stamp: "Formatted at 2024-05-01 20:17:49 on dist-test-slave-pg4x"
I20240501 20:17:49.585984 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ee984883adca4b1fada1f685b5275ca0"
format_stamp: "Formatted at 2024-05-01 20:17:49 on dist-test-slave-pg4x"
I20240501 20:17:49.591223 14914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:17:49.594664 14928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:49.595957 14914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:17:49.596091 14914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ee984883adca4b1fada1f685b5275ca0"
format_stamp: "Formatted at 2024-05-01 20:17:49 on dist-test-slave-pg4x"
I20240501 20:17:49.596264 14914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:49.613425 14914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:49.614212 14914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:49.614460 14914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:49.615413 14914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:49.616860 14914 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:49.616947 14914 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:49.617035 14914 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:49.617091 14914 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:49.649932 14914 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.193:46715
I20240501 20:17:49.650007 15040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.193:46715 every 8 connection(s)
I20240501 20:17:49.651314 14914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:17:49.661042 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 14914
I20240501 20:17:49.661296 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:17:49.664837 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.194:0
--local_ip_for_outbound_sockets=127.14.106.194
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--builtin_ntp_servers=127.14.106.212:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:17:49.665110 15041 heartbeater.cc:347] Connected to a master server at 127.14.106.254:39919
I20240501 20:17:49.665386 15041 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:49.666196 15041 heartbeater.cc:511] Master 127.14.106.254:39919 requested a full tablet report, sending...
I20240501 20:17:49.751461 14855 ts_manager.cc:194] Registered new tserver with Master: ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715)
I20240501 20:17:49.753298 14855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.193:34905
W20240501 20:17:50.019930 15046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:50.020188 15046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:50.020275 15046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:50.027415 15046 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:17:50.027532 15046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:50.027716 15046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.194
I20240501 20:17:50.036494 15046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.194
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:50.037771 15046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:50.039458 15046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:50.050905 15051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:50.051343 15052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:50.052011 15054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:50.052759 15046 server_base.cc:1034] running on GCE node
I20240501 20:17:50.053364 15046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:50.054235 15046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:50.055424 15046 hybrid_clock.cc:648] HybridClock initialized: now 1714594670055382 us; error 37 us; skew 500 ppm
I20240501 20:17:50.055718 15046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:50.057767 15046 webserver.cc:460] Webserver started at http://127.14.106.194:38835/ using document root <none> and password file <none>
I20240501 20:17:50.058435 15046 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:50.058571 15046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:50.058884 15046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:50.061358 15046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:50.062124 15046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:50.067260 15046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:17:50.070485 15060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:50.071753 15046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:50.071909 15046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:50.072106 15046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:50.101958 15046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:50.102820 15046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:50.103080 15046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:50.104014 15046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:50.105443 15046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:50.105538 15046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:50.105681 15046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:50.105759 15046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:50.137424 15046 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.194:41097
I20240501 20:17:50.137518 15172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.194:41097 every 8 connection(s)
I20240501 20:17:50.138725 15046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:17:50.142853 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15046
I20240501 20:17:50.143018 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:17:50.146209 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.195:0
--local_ip_for_outbound_sockets=127.14.106.195
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--builtin_ntp_servers=127.14.106.212:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:17:50.151336 15173 heartbeater.cc:347] Connected to a master server at 127.14.106.254:39919
I20240501 20:17:50.151616 15173 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:50.152464 15173 heartbeater.cc:511] Master 127.14.106.254:39919 requested a full tablet report, sending...
I20240501 20:17:50.196676 14855 ts_manager.cc:194] Registered new tserver with Master: e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097)
I20240501 20:17:50.197616 14855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.194:46821
W20240501 20:17:50.482084 15176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:50.482371 15176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:50.482515 15176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:50.489665 15176 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:17:50.489776 15176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:50.489946 15176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.195
I20240501 20:17:50.498818 15176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.195
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:50.500074 15176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:50.501765 15176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:50.513718 15185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:50.514222 15176 server_base.cc:1034] running on GCE node
W20240501 20:17:50.514752 15182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:50.515308 15183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:50.516003 15176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:50.516901 15176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:50.518136 15176 hybrid_clock.cc:648] HybridClock initialized: now 1714594670518076 us; error 55 us; skew 500 ppm
I20240501 20:17:50.518471 15176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:50.525571 15176 webserver.cc:460] Webserver started at http://127.14.106.195:40723/ using document root <none> and password file <none>
I20240501 20:17:50.526343 15176 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:50.526473 15176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:50.526839 15176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:50.529520 15176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0021804537544af482a273d050ee9765"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:50.530277 15176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0021804537544af482a273d050ee9765"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:50.535377 15176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:17:50.538903 15191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:50.540160 15176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:17:50.540318 15176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0021804537544af482a273d050ee9765"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:50.540524 15176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:50.557893 15176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:50.558813 15176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:50.559105 15176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:50.560146 15176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:50.561781 15176 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:50.561898 15176 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:50.562021 15176 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:50.562098 15176 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:50.594547 15176 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.195:45129
I20240501 20:17:50.594611 15303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.195:45129 every 8 connection(s)
I20240501 20:17:50.595867 15176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:17:50.598839 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15176
I20240501 20:17:50.599035 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:17:50.603222 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:0
--local_ip_for_outbound_sockets=127.14.106.196
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--builtin_ntp_servers=127.14.106.212:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:17:50.609813 15304 heartbeater.cc:347] Connected to a master server at 127.14.106.254:39919
I20240501 20:17:50.610119 15304 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:50.610991 15304 heartbeater.cc:511] Master 127.14.106.254:39919 requested a full tablet report, sending...
I20240501 20:17:50.666811 14855 ts_manager.cc:194] Registered new tserver with Master: 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:50.668078 14855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.195:43431
I20240501 20:17:50.756407 15041 heartbeater.cc:503] Master 127.14.106.254:39919 was elected leader, sending a full tablet report...
W20240501 20:17:50.956487 15307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:50.956753 15307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:50.956846 15307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:50.964157 15307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:17:50.964282 15307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:50.964406 15307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:17:50.973155 15307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:50.974381 15307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:50.976084 15307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:50.987749 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:50.988204 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:50.989683 15307 server_base.cc:1034] running on GCE node
W20240501 20:17:50.989725 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:50.990365 15307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:50.991339 15307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:50.992547 15307 hybrid_clock.cc:648] HybridClock initialized: now 1714594670992501 us; error 36 us; skew 500 ppm
I20240501 20:17:50.992853 15307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:50.995155 15307 webserver.cc:460] Webserver started at http://127.14.106.196:36177/ using document root <none> and password file <none>
I20240501 20:17:50.995860 15307 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:50.995976 15307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:50.996276 15307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:50.998901 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "207d3f61d06541f5ae21e329b9fb5f8e"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:50.999637 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "207d3f61d06541f5ae21e329b9fb5f8e"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:51.004873 15307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:17:51.008169 15322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:51.009397 15307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:17:51.009567 15307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "207d3f61d06541f5ae21e329b9fb5f8e"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:17:51.009766 15307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:51.034037 15307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:51.034967 15307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:51.035240 15307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:51.036211 15307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:51.037859 15307 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:51.037961 15307 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:51.038098 15307 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:51.038172 15307 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:51.070937 15307 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:35173
I20240501 20:17:51.071007 15434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:35173 every 8 connection(s)
I20240501 20:17:51.072284 15307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:17:51.073838 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15307
I20240501 20:17:51.073997 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:17:51.077720 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:0
--local_ip_for_outbound_sockets=127.14.106.197
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--builtin_ntp_servers=127.14.106.212:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:17:51.087795 15435 heartbeater.cc:347] Connected to a master server at 127.14.106.254:39919
I20240501 20:17:51.088137 15435 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:51.089157 15435 heartbeater.cc:511] Master 127.14.106.254:39919 requested a full tablet report, sending...
I20240501 20:17:51.139079 14855 ts_manager.cc:194] Registered new tserver with Master: 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196:35173)
I20240501 20:17:51.140043 14855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:50335
I20240501 20:17:51.200450 15173 heartbeater.cc:503] Master 127.14.106.254:39919 was elected leader, sending a full tablet report...
W20240501 20:17:51.431854 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:51.432163 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:51.432312 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:51.439754 15438 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:17:51.439899 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:51.440071 15438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:17:51.449143 15438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=0
--tserver_master_addrs=127.14.106.254:39919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:51.450441 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:51.452162 15438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:51.463980 15445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:51.464689 15444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:51.465368 15447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:51.466001 15438 server_base.cc:1034] running on GCE node
I20240501 20:17:51.466581 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:51.467535 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:51.468725 15438 hybrid_clock.cc:648] HybridClock initialized: now 1714594671468681 us; error 39 us; skew 500 ppm
I20240501 20:17:51.469027 15438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:51.471302 15438 webserver.cc:460] Webserver started at http://127.14.106.197:40283/ using document root <none> and password file <none>
I20240501 20:17:51.472023 15438 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:51.472146 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:51.472450 15438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:17:51.475100 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "79268a903afc4ef3a3bd79a32174fe2e"
format_stamp: "Formatted at 2024-05-01 20:17:51 on dist-test-slave-pg4x"
I20240501 20:17:51.475900 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "79268a903afc4ef3a3bd79a32174fe2e"
format_stamp: "Formatted at 2024-05-01 20:17:51 on dist-test-slave-pg4x"
I20240501 20:17:51.481129 15438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:17:51.484567 15453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:51.485864 15438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:17:51.486021 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "79268a903afc4ef3a3bd79a32174fe2e"
format_stamp: "Formatted at 2024-05-01 20:17:51 on dist-test-slave-pg4x"
I20240501 20:17:51.486230 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:51.511574 15438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:51.512399 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:51.512666 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:51.513633 15438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:51.515136 15438 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:51.515234 15438 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:51.515353 15438 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:51.515425 15438 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:51.546973 15438 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:41431
I20240501 20:17:51.547039 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:41431 every 8 connection(s)
I20240501 20:17:51.548254 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:17:51.558552 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15438
I20240501 20:17:51.558773 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:17:51.560966 15566 heartbeater.cc:347] Connected to a master server at 127.14.106.254:39919
I20240501 20:17:51.561252 15566 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:51.562052 15566 heartbeater.cc:511] Master 127.14.106.254:39919 requested a full tablet report, sending...
I20240501 20:17:51.608911 14854 ts_manager.cc:194] Registered new tserver with Master: 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197:41431)
I20240501 20:17:51.609933 14854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:52161
I20240501 20:17:51.612910 14763 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:17:51.629715 14763 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:17:51.629855 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15438
I20240501 20:17:51.638070 15561 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:51.671320 15304 heartbeater.cc:503] Master 127.14.106.254:39919 was elected leader, sending a full tablet report...
I20240501 20:17:51.755458 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15438
I20240501 20:17:51.773926 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15307
I20240501 20:17:51.782109 15430 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:17:51.864421 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15307
I20240501 20:17:55.633889 14854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47876:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:17:55.640763 14854 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:17:55.683689 14974 tablet_service.cc:1467] Processing CreateTablet for tablet c134aed0124e4f35a57368d5855a0ff9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.684039 15108 tablet_service.cc:1467] Processing CreateTablet for tablet c32bb5190ebc49b5ad35a025746fa85d (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.684039 15239 tablet_service.cc:1467] Processing CreateTablet for tablet c32bb5190ebc49b5ad35a025746fa85d (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.684978 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c134aed0124e4f35a57368d5855a0ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.685365 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32bb5190ebc49b5ad35a025746fa85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.685365 15239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32bb5190ebc49b5ad35a025746fa85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.685853 15106 tablet_service.cc:1467] Processing CreateTablet for tablet c134aed0124e4f35a57368d5855a0ff9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.686810 15106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c134aed0124e4f35a57368d5855a0ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.687079 15237 tablet_service.cc:1467] Processing CreateTablet for tablet c134aed0124e4f35a57368d5855a0ff9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.686197 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 23764dcc72b34239876036a7b99a6b6d (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.687800 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 23764dcc72b34239876036a7b99a6b6d (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.687953 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23764dcc72b34239876036a7b99a6b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.688294 14975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23764dcc72b34239876036a7b99a6b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.689165 14976 tablet_service.cc:1467] Processing CreateTablet for tablet c32bb5190ebc49b5ad35a025746fa85d (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.689689 14976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32bb5190ebc49b5ad35a025746fa85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.691944 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 23764dcc72b34239876036a7b99a6b6d (DEFAULT_TABLE table=rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:55.700325 15237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c134aed0124e4f35a57368d5855a0ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.702975 15238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23764dcc72b34239876036a7b99a6b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:55.706032 15591 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:55.709697 15591 tablet_bootstrap.cc:654] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.711102 15591 log.cc:824] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log is configured to *not* fsync() on all Append() calls
I20240501 20:17:55.714408 15590 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:55.718909 15590 tablet_bootstrap.cc:654] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.719708 15593 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:55.721220 15590 log.cc:824] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:17:55.723320 15593 tablet_bootstrap.cc:654] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.724699 15593 log.cc:824] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765: Log is configured to *not* fsync() on all Append() calls
I20240501 20:17:55.728155 15591 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:55.728473 15593 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:55.728462 15591 ts_tablet_manager.cc:1367] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.008s
I20240501 20:17:55.728674 15590 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:55.728772 15593 ts_tablet_manager.cc:1367] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 20:17:55.728928 15590 ts_tablet_manager.cc:1367] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20240501 20:17:55.734870 15591 raft_consensus.cc:348] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.735219 15591 raft_consensus.cc:374] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.735368 15591 raft_consensus.cc:729] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.736021 15591 consensus_queue.cc:260] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.737709 15591 ts_tablet_manager.cc:1398] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240501 20:17:55.737816 15593 raft_consensus.cc:348] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.737816 15590 raft_consensus.cc:348] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.738144 15591 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:55.738232 15590 raft_consensus.cc:374] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.738312 15593 raft_consensus.cc:374] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.738406 15590 raft_consensus.cc:729] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.738457 15593 raft_consensus.cc:729] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.739228 15590 consensus_queue.cc:260] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.739229 15593 consensus_queue.cc:260] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.739784 15591 tablet_bootstrap.cc:654] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.746057 15591 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:55.746213 15591 ts_tablet_manager.cc:1367] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240501 20:17:55.747438 15590 ts_tablet_manager.cc:1398] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20240501 20:17:55.747251 15591 raft_consensus.cc:348] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.747665 15591 raft_consensus.cc:374] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.747788 15591 raft_consensus.cc:729] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.747866 15590 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:55.748078 15591 consensus_queue.cc:260] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.748654 15591 ts_tablet_manager.cc:1398] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:17:55.748946 15591 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:55.749517 15590 tablet_bootstrap.cc:654] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.749585 15593 ts_tablet_manager.cc:1398] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.021s user 0.012s sys 0.005s
I20240501 20:17:55.750536 15591 tablet_bootstrap.cc:654] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.750599 15593 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:55.754277 15593 tablet_bootstrap.cc:654] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.757835 15590 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:55.757992 15590 ts_tablet_manager.cc:1367] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240501 20:17:55.758183 15591 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:55.758430 15591 ts_tablet_manager.cc:1367] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20240501 20:17:55.758827 15593 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:55.758985 15593 ts_tablet_manager.cc:1367] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240501 20:17:55.758884 15590 raft_consensus.cc:348] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.759076 15590 raft_consensus.cc:374] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.759178 15590 raft_consensus.cc:729] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.759272 15591 raft_consensus.cc:348] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.759416 15590 consensus_queue.cc:260] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.759522 15591 raft_consensus.cc:374] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.759603 15591 raft_consensus.cc:729] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.759934 15590 ts_tablet_manager.cc:1398] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:17:55.759910 15591 consensus_queue.cc:260] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.760195 15590 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:55.760488 15591 ts_tablet_manager.cc:1398] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:17:55.761163 15593 raft_consensus.cc:348] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.761394 15593 raft_consensus.cc:374] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.761514 15593 raft_consensus.cc:729] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.761816 15590 tablet_bootstrap.cc:654] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.761763 15593 consensus_queue.cc:260] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.762347 15593 ts_tablet_manager.cc:1398] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:17:55.762751 15593 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:55.764339 15593 tablet_bootstrap.cc:654] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:55.766096 15593 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:55.766243 15593 ts_tablet_manager.cc:1367] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:17:55.766909 15593 raft_consensus.cc:348] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.767078 15593 raft_consensus.cc:374] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.767154 15593 raft_consensus.cc:729] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.767372 15593 consensus_queue.cc:260] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.767655 15590 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:55.767807 15590 ts_tablet_manager.cc:1367] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240501 20:17:55.767867 15593 ts_tablet_manager.cc:1398] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:17:55.768430 15590 raft_consensus.cc:348] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.768689 15590 raft_consensus.cc:374] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:55.768786 15590 raft_consensus.cc:729] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:55.769054 15590 consensus_queue.cc:260] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.769641 15590 ts_tablet_manager.cc:1398] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:17:55.814518 15596 raft_consensus.cc:482] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:55.814813 15596 raft_consensus.cc:504] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.816524 15596 leader_election.cc:288] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:55.823554 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c134aed0124e4f35a57368d5855a0ff9" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0" is_pre_election: true
I20240501 20:17:55.823801 15259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c134aed0124e4f35a57368d5855a0ff9" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765" is_pre_election: true
I20240501 20:17:55.824043 14996 raft_consensus.cc:2454] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 0.
I20240501 20:17:55.824287 15259 raft_consensus.cc:2454] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 0.
I20240501 20:17:55.824636 15061 leader_election.cc:302] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e41b3aa39c7b4b0c90f3c36d6f07a0b4, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:55.825160 15596 raft_consensus.cc:2781] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:55.825356 15596 raft_consensus.cc:482] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:55.825464 15596 raft_consensus.cc:3037] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:55.827565 15596 raft_consensus.cc:504] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.828150 15596 leader_election.cc:288] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 election: Requested vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:55.828522 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c134aed0124e4f35a57368d5855a0ff9" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0"
I20240501 20:17:55.828635 15259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c134aed0124e4f35a57368d5855a0ff9" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765"
I20240501 20:17:55.828696 14996 raft_consensus.cc:3037] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:55.828838 15259 raft_consensus.cc:3037] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:55.830754 14996 raft_consensus.cc:2454] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 1.
I20240501 20:17:55.830943 15259 raft_consensus.cc:2454] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 1.
I20240501 20:17:55.831153 15061 leader_election.cc:302] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e41b3aa39c7b4b0c90f3c36d6f07a0b4, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:55.831441 15596 raft_consensus.cc:2781] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:55.831868 15596 raft_consensus.cc:686] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Becoming Leader. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Running, Role: LEADER
I20240501 20:17:55.832386 15596 consensus_queue.cc:237] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.836941 14854 catalog_manager.cc:5484] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: UNKNOWN } } }
W20240501 20:17:55.851303 15305 tablet.cc:2369] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:17:55.894532 15174 tablet.cc:2369] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:17:55.907506 15042 tablet.cc:2369] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:17:55.949723 15598 raft_consensus.cc:482] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:55.949939 15598 raft_consensus.cc:504] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.951606 15598 leader_election.cc:288] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:55.963171 15259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23764dcc72b34239876036a7b99a6b6d" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765" is_pre_election: true
I20240501 20:17:55.963505 15259 raft_consensus.cc:2454] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 0.
I20240501 20:17:55.963507 15128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23764dcc72b34239876036a7b99a6b6d" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" is_pre_election: true
I20240501 20:17:55.963963 15128 raft_consensus.cc:2454] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 0.
I20240501 20:17:55.964021 14932 leader_election.cc:302] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:55.964469 15598 raft_consensus.cc:2781] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:55.964607 15598 raft_consensus.cc:482] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:55.964709 15598 raft_consensus.cc:3037] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:55.966727 15598 raft_consensus.cc:504] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.967298 15598 leader_election.cc:288] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 election: Requested vote from peers e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:55.967732 15259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23764dcc72b34239876036a7b99a6b6d" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765"
I20240501 20:17:55.967743 15128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23764dcc72b34239876036a7b99a6b6d" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
I20240501 20:17:55.967917 15128 raft_consensus.cc:3037] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:55.967921 15259 raft_consensus.cc:3037] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:55.969739 15259 raft_consensus.cc:2454] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 1.
I20240501 20:17:55.969833 15128 raft_consensus.cc:2454] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 1.
I20240501 20:17:55.970160 14932 leader_election.cc:302] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:55.970441 15598 raft_consensus.cc:2781] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:55.970871 15598 raft_consensus.cc:686] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Becoming Leader. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Running, Role: LEADER
I20240501 20:17:55.971324 15598 consensus_queue.cc:237] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:55.975672 14854 catalog_manager.cc:5484] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee984883adca4b1fada1f685b5275ca0 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: UNKNOWN } } }
I20240501 20:17:56.007380 15597 raft_consensus.cc:482] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:56.007562 15597 raft_consensus.cc:504] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.009122 15597 leader_election.cc:288] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715)
I20240501 20:17:56.015774 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32bb5190ebc49b5ad35a025746fa85d" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0" is_pre_election: true
I20240501 20:17:56.015931 15128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32bb5190ebc49b5ad35a025746fa85d" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" is_pre_election: true
I20240501 20:17:56.016018 14996 raft_consensus.cc:2454] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 0.
I20240501 20:17:56.016167 15128 raft_consensus.cc:2454] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 0.
I20240501 20:17:56.016535 15192 leader_election.cc:302] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:56.016959 15597 raft_consensus.cc:2781] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:56.017077 15597 raft_consensus.cc:482] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:56.017176 15597 raft_consensus.cc:3037] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:56.019269 15597 raft_consensus.cc:504] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.019954 15597 leader_election.cc:288] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 election: Requested vote from peers e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715)
I20240501 20:17:56.020385 15128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32bb5190ebc49b5ad35a025746fa85d" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
I20240501 20:17:56.020432 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32bb5190ebc49b5ad35a025746fa85d" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0"
I20240501 20:17:56.020571 15128 raft_consensus.cc:3037] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:56.020673 14996 raft_consensus.cc:3037] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:56.022516 15128 raft_consensus.cc:2454] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 1.
I20240501 20:17:56.022598 14996 raft_consensus.cc:2454] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 1.
I20240501 20:17:56.022982 15193 leader_election.cc:302] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, e41b3aa39c7b4b0c90f3c36d6f07a0b4; no voters:
I20240501 20:17:56.023252 15597 raft_consensus.cc:2781] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:56.023612 15597 raft_consensus.cc:686] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Becoming Leader. State: Replica: 0021804537544af482a273d050ee9765, State: Running, Role: LEADER
I20240501 20:17:56.024144 15597 consensus_queue.cc:237] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.028470 14854 catalog_manager.cc:5484] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0021804537544af482a273d050ee9765 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } }
I20240501 20:17:56.227831 15596 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:56.237129 15596 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:56.398157 15598 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:56.403519 15598 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:17:56.458742 15617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:56.459046 15617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:56.466722 15617 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 20:17:56.557581 15597 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:56.565685 15597 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 244 ms
time per row: 0.122 ms
I20240501 20:17:56.854988 14847 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47876:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:17:56.855754 14847 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:17:56.864652 14976 tablet_service.cc:1467] Processing CreateTablet for tablet 110b4b780c48418fbd1dc18c489c5fce (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.864879 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 110b4b780c48418fbd1dc18c489c5fce (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.865201 14976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b4b780c48418fbd1dc18c489c5fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.865376 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b4b780c48418fbd1dc18c489c5fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.865406 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 3822c87b8cd44c4dbf432adf9aa0f590 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.865731 15106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3822c87b8cd44c4dbf432adf9aa0f590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.866413 15237 tablet_service.cc:1467] Processing CreateTablet for tablet 110b4b780c48418fbd1dc18c489c5fce (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.867076 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 3822c87b8cd44c4dbf432adf9aa0f590 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.868197 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 3822c87b8cd44c4dbf432adf9aa0f590 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.868607 14975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3822c87b8cd44c4dbf432adf9aa0f590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.874253 14974 tablet_service.cc:1467] Processing CreateTablet for tablet f062277a14ab4a6a83dbc7b1bc49eab4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.874861 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f062277a14ab4a6a83dbc7b1bc49eab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.876381 15680 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:56.878680 15239 tablet_service.cc:1467] Processing CreateTablet for tablet f062277a14ab4a6a83dbc7b1bc49eab4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.879603 15680 tablet_bootstrap.cc:654] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.879770 15238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3822c87b8cd44c4dbf432adf9aa0f590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.880084 15107 tablet_service.cc:1467] Processing CreateTablet for tablet f062277a14ab4a6a83dbc7b1bc49eab4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:56.880616 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f062277a14ab4a6a83dbc7b1bc49eab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.883569 15237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b4b780c48418fbd1dc18c489c5fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.889143 15239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f062277a14ab4a6a83dbc7b1bc49eab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:56.889503 15679 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:56.892122 15679 tablet_bootstrap.cc:654] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.893957 15682 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:56.895921 15682 tablet_bootstrap.cc:654] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.904356 15682 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:56.904618 15682 ts_tablet_manager.cc:1367] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 20:17:56.905550 15682 raft_consensus.cc:348] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.905913 15682 raft_consensus.cc:374] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.906059 15682 raft_consensus.cc:729] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.906404 15682 consensus_queue.cc:260] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.908550 15682 ts_tablet_manager.cc:1398] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:17:56.908944 15682 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:56.910357 15680 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:56.910382 15679 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:56.910534 15680 ts_tablet_manager.cc:1367] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.019s
I20240501 20:17:56.910732 15679 ts_tablet_manager.cc:1367] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20240501 20:17:56.911029 15682 tablet_bootstrap.cc:654] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.911540 15680 raft_consensus.cc:348] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.911787 15680 raft_consensus.cc:374] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.911913 15680 raft_consensus.cc:729] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.911723 15679 raft_consensus.cc:348] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.912196 15679 raft_consensus.cc:374] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.912343 15679 raft_consensus.cc:729] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.912259 15680 consensus_queue.cc:260] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.912986 15680 ts_tablet_manager.cc:1398] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:17:56.912848 15679 consensus_queue.cc:260] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.913277 15680 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:56.914166 15679 ts_tablet_manager.cc:1398] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:17:56.914537 15679 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:56.915458 15680 tablet_bootstrap.cc:654] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.916425 15679 tablet_bootstrap.cc:654] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.921931 15682 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:56.922117 15682 ts_tablet_manager.cc:1367] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240501 20:17:56.922215 15679 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:56.922425 15679 ts_tablet_manager.cc:1367] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240501 20:17:56.922457 15680 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:56.922652 15680 ts_tablet_manager.cc:1367] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20240501 20:17:56.923049 15682 raft_consensus.cc:348] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.923303 15682 raft_consensus.cc:374] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.923239 15679 raft_consensus.cc:348] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.923393 15682 raft_consensus.cc:729] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.923476 15679 raft_consensus.cc:374] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.923552 15679 raft_consensus.cc:729] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.923442 15680 raft_consensus.cc:348] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.923669 15680 raft_consensus.cc:374] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.923756 15680 raft_consensus.cc:729] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.923661 15682 consensus_queue.cc:260] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.923810 15679 consensus_queue.cc:260] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.924017 15680 consensus_queue.cc:260] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.924197 15682 ts_tablet_manager.cc:1398] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:17:56.924576 15679 ts_tablet_manager.cc:1398] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:17:56.924582 15680 ts_tablet_manager.cc:1398] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:17:56.924880 15680 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:56.924882 15679 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:56.926587 15679 tablet_bootstrap.cc:654] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.926594 15680 tablet_bootstrap.cc:654] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.931027 15682 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:56.932327 15682 tablet_bootstrap.cc:654] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:56.935060 15680 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:56.935259 15680 ts_tablet_manager.cc:1367] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:17:56.935976 15679 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:56.936282 15679 ts_tablet_manager.cc:1367] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240501 20:17:56.936218 15680 raft_consensus.cc:348] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.936443 15680 raft_consensus.cc:374] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.936540 15682 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:56.936543 15680 raft_consensus.cc:729] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.936692 15682 ts_tablet_manager.cc:1367] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 20:17:56.936828 15680 consensus_queue.cc:260] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.937544 15680 ts_tablet_manager.cc:1398] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:17:56.937475 15682 raft_consensus.cc:348] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.937687 15682 raft_consensus.cc:374] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.937788 15682 raft_consensus.cc:729] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.938050 15682 consensus_queue.cc:260] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.938593 15682 ts_tablet_manager.cc:1398] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:17:56.938694 15679 raft_consensus.cc:348] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.939532 15679 raft_consensus.cc:374] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:56.939675 15679 raft_consensus.cc:729] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:56.939930 15679 consensus_queue.cc:260] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:56.940618 15679 ts_tablet_manager.cc:1398] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:17:57.054682 15607 raft_consensus.cc:482] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:57.054926 15607 raft_consensus.cc:504] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.055711 15607 leader_election.cc:288] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:57.056414 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0" is_pre_election: true
I20240501 20:17:57.056392 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765" is_pre_election: true
I20240501 20:17:57.056644 15258 raft_consensus.cc:2454] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 0.
I20240501 20:17:57.056644 14996 raft_consensus.cc:2454] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 0.
I20240501 20:17:57.057183 15064 leader_election.cc:302] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, e41b3aa39c7b4b0c90f3c36d6f07a0b4; no voters:
I20240501 20:17:57.057461 15607 raft_consensus.cc:2781] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:57.057571 15607 raft_consensus.cc:482] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:57.057665 15607 raft_consensus.cc:3037] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.059644 15607 raft_consensus.cc:504] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.060173 15607 leader_election.cc:288] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 election: Requested vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:57.060659 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765"
I20240501 20:17:57.060734 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0"
I20240501 20:17:57.060855 15258 raft_consensus.cc:3037] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.060954 14996 raft_consensus.cc:3037] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.062821 14996 raft_consensus.cc:2454] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 1.
I20240501 20:17:57.062837 15258 raft_consensus.cc:2454] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 1.
I20240501 20:17:57.063221 15064 leader_election.cc:302] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, e41b3aa39c7b4b0c90f3c36d6f07a0b4; no voters:
I20240501 20:17:57.063541 15607 raft_consensus.cc:2781] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:57.063660 15607 raft_consensus.cc:686] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Becoming Leader. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Running, Role: LEADER
I20240501 20:17:57.063964 15607 consensus_queue.cc:237] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.066679 14847 catalog_manager.cc:5484] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: UNKNOWN } } }
I20240501 20:17:57.144937 15650 raft_consensus.cc:482] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:57.145139 15650 raft_consensus.cc:504] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.146014 15650 leader_election.cc:288] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097)
I20240501 20:17:57.146433 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b4b780c48418fbd1dc18c489c5fce" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0" is_pre_election: true
I20240501 20:17:57.146664 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b4b780c48418fbd1dc18c489c5fce" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" is_pre_election: true
I20240501 20:17:57.146754 14996 raft_consensus.cc:2454] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 0.
I20240501 20:17:57.146925 15127 raft_consensus.cc:2454] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 0.
I20240501 20:17:57.147269 15192 leader_election.cc:302] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:57.147609 15650 raft_consensus.cc:2781] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:57.147740 15650 raft_consensus.cc:482] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:57.147825 15650 raft_consensus.cc:3037] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.149817 15650 raft_consensus.cc:504] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.150409 15650 leader_election.cc:288] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 election: Requested vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097)
I20240501 20:17:57.150825 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b4b780c48418fbd1dc18c489c5fce" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0"
I20240501 20:17:57.150960 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b4b780c48418fbd1dc18c489c5fce" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
I20240501 20:17:57.151032 14996 raft_consensus.cc:3037] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.151162 15127 raft_consensus.cc:3037] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.153087 14996 raft_consensus.cc:2454] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 1.
I20240501 20:17:57.153493 15192 leader_election.cc:302] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:57.153751 15650 raft_consensus.cc:2781] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:57.153908 15650 raft_consensus.cc:686] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Becoming Leader. State: Replica: 0021804537544af482a273d050ee9765, State: Running, Role: LEADER
I20240501 20:17:57.154106 15127 raft_consensus.cc:2454] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 1.
I20240501 20:17:57.154212 15650 consensus_queue.cc:237] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.157409 14847 catalog_manager.cc:5484] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0021804537544af482a273d050ee9765 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } }
I20240501 20:17:57.168481 15598 raft_consensus.cc:482] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:57.168659 15598 raft_consensus.cc:504] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.169220 15598 leader_election.cc:288] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:57.169660 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f062277a14ab4a6a83dbc7b1bc49eab4" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765" is_pre_election: true
I20240501 20:17:57.169685 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f062277a14ab4a6a83dbc7b1bc49eab4" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" is_pre_election: true
I20240501 20:17:57.169883 15258 raft_consensus.cc:2454] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 0.
I20240501 20:17:57.169888 15127 raft_consensus.cc:2454] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 0.
I20240501 20:17:57.170284 14930 leader_election.cc:302] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, e41b3aa39c7b4b0c90f3c36d6f07a0b4, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:57.170562 15598 raft_consensus.cc:2781] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:57.170686 15598 raft_consensus.cc:482] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:57.170799 15598 raft_consensus.cc:3037] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.172737 15598 raft_consensus.cc:504] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.173317 15598 leader_election.cc:288] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 election: Requested vote from peers e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:57.173712 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f062277a14ab4a6a83dbc7b1bc49eab4" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
I20240501 20:17:57.173797 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f062277a14ab4a6a83dbc7b1bc49eab4" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765"
I20240501 20:17:57.173945 15127 raft_consensus.cc:3037] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.173985 15258 raft_consensus.cc:3037] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:57.176959 15258 raft_consensus.cc:2454] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 1.
I20240501 20:17:57.176964 15127 raft_consensus.cc:2454] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 1.
I20240501 20:17:57.177368 14930 leader_election.cc:302] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e41b3aa39c7b4b0c90f3c36d6f07a0b4, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:57.177680 15598 raft_consensus.cc:2781] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:57.177855 15598 raft_consensus.cc:686] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Becoming Leader. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Running, Role: LEADER
I20240501 20:17:57.178105 15598 consensus_queue.cc:237] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:57.180913 14847 catalog_manager.cc:5484] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee984883adca4b1fada1f685b5275ca0 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: UNKNOWN } } }
I20240501 20:17:57.567435 15607 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:57.574550 15696 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:17:57.594550 15694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:57.594923 15694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:57.602612 15694 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:17:57.620155 15598 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:57.628727 15598 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 20:17:57.659497 15650 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:57.669941 15650 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 226.191 ms
time per row: 0.113096 ms
I20240501 20:17:57.977993 14855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47876:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:17:57.978792 14855 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:17:57.987180 14974 tablet_service.cc:1467] Processing CreateTablet for tablet b82e8ea1f3484534ae16e968fab04860 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.987248 15107 tablet_service.cc:1467] Processing CreateTablet for tablet b82e8ea1f3484534ae16e968fab04860 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.987648 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82e8ea1f3484534ae16e968fab04860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:57.987708 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82e8ea1f3484534ae16e968fab04860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:57.988955 15238 tablet_service.cc:1467] Processing CreateTablet for tablet b82e8ea1f3484534ae16e968fab04860 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.988994 15239 tablet_service.cc:1467] Processing CreateTablet for tablet 925816a53f1e45a4a81efd7dfddd2b9d (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.989506 15238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82e8ea1f3484534ae16e968fab04860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:57.989531 15237 tablet_service.cc:1467] Processing CreateTablet for tablet 682d92d873304a308fc82d94081f3776 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.989863 15237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682d92d873304a308fc82d94081f3776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:57.990123 15239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925816a53f1e45a4a81efd7dfddd2b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:57.994119 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 682d92d873304a308fc82d94081f3776 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.994539 15106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682d92d873304a308fc82d94081f3776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:57.994988 15108 tablet_service.cc:1467] Processing CreateTablet for tablet 925816a53f1e45a4a81efd7dfddd2b9d (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.996053 14976 tablet_service.cc:1467] Processing CreateTablet for tablet 682d92d873304a308fc82d94081f3776 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.996178 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 925816a53f1e45a4a81efd7dfddd2b9d (DEFAULT_TABLE table=rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:17:57.996477 14976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682d92d873304a308fc82d94081f3776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:57.996758 14975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925816a53f1e45a4a81efd7dfddd2b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:57.999464 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925816a53f1e45a4a81efd7dfddd2b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:17:58.008838 15759 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:58.011318 15759 tablet_bootstrap.cc:654] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.011735 15760 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:58.014073 15760 tablet_bootstrap.cc:654] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.016693 15760 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:58.016784 15758 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:58.016844 15760 ts_tablet_manager.cc:1367] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240501 20:17:58.017282 15759 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:58.017427 15759 ts_tablet_manager.cc:1367] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240501 20:17:58.017642 15760 raft_consensus.cc:348] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.017838 15760 raft_consensus.cc:374] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.017925 15760 raft_consensus.cc:729] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.018177 15760 consensus_queue.cc:260] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.018239 15759 raft_consensus.cc:348] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.018422 15759 raft_consensus.cc:374] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.018594 15759 raft_consensus.cc:729] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.018791 15758 tablet_bootstrap.cc:654] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.018908 15760 ts_tablet_manager.cc:1398] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:17:58.018934 15759 consensus_queue.cc:260] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.019197 15760 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:58.019493 15759 ts_tablet_manager.cc:1398] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:17:58.019745 15759 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:58.020872 15760 tablet_bootstrap.cc:654] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.021313 15759 tablet_bootstrap.cc:654] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.029006 15760 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:58.029208 15760 ts_tablet_manager.cc:1367] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240501 20:17:58.029534 15758 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:58.029688 15758 ts_tablet_manager.cc:1367] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240501 20:17:58.030115 15760 raft_consensus.cc:348] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.030382 15760 raft_consensus.cc:374] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.030480 15760 raft_consensus.cc:729] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.030776 15760 consensus_queue.cc:260] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.030902 15759 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:58.030772 15758 raft_consensus.cc:348] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.031064 15758 raft_consensus.cc:374] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.031176 15758 raft_consensus.cc:729] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.031440 15760 ts_tablet_manager.cc:1398] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:17:58.031416 15758 consensus_queue.cc:260] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.031737 15760 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:17:58.032011 15759 ts_tablet_manager.cc:1367] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240501 20:17:58.032089 15758 ts_tablet_manager.cc:1398] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:17:58.032392 15758 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:58.033383 15760 tablet_bootstrap.cc:654] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.033990 15758 tablet_bootstrap.cc:654] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.035315 15759 raft_consensus.cc:348] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.035657 15759 raft_consensus.cc:374] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.035742 15759 raft_consensus.cc:729] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.036044 15759 consensus_queue.cc:260] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.036679 15759 ts_tablet_manager.cc:1398] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20240501 20:17:58.037230 15759 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:17:58.039484 15759 tablet_bootstrap.cc:654] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.041882 15760 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0: No bootstrap required, opened a new log
I20240501 20:17:58.042043 15760 ts_tablet_manager.cc:1367] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 20:17:58.042937 15760 raft_consensus.cc:348] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.043151 15760 raft_consensus.cc:374] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.043248 15760 raft_consensus.cc:729] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.043542 15760 consensus_queue.cc:260] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.044252 15760 ts_tablet_manager.cc:1398] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:17:58.045712 15758 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:58.045919 15758 ts_tablet_manager.cc:1367] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240501 20:17:58.046255 15759 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: No bootstrap required, opened a new log
I20240501 20:17:58.046416 15759 ts_tablet_manager.cc:1367] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:17:58.046801 15758 raft_consensus.cc:348] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.047049 15758 raft_consensus.cc:374] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.047148 15758 raft_consensus.cc:729] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.047209 15759 raft_consensus.cc:348] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.047425 15759 raft_consensus.cc:374] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.047521 15759 raft_consensus.cc:729] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.047427 15758 consensus_queue.cc:260] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.047752 15759 consensus_queue.cc:260] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.048285 15759 ts_tablet_manager.cc:1398] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:17:58.048774 15758 ts_tablet_manager.cc:1398] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:58.049062 15758 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:17:58.051590 15758 tablet_bootstrap.cc:654] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Neither blocks nor log segments found. Creating new log.
I20240501 20:17:58.053617 15758 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: No bootstrap required, opened a new log
I20240501 20:17:58.053728 15758 ts_tablet_manager.cc:1367] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 20:17:58.054265 15758 raft_consensus.cc:348] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.054454 15758 raft_consensus.cc:374] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:17:58.054523 15758 raft_consensus.cc:729] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: FOLLOWER
I20240501 20:17:58.054734 15758 consensus_queue.cc:260] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.055177 15758 ts_tablet_manager.cc:1398] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 20:17:58.132272 15699 raft_consensus.cc:482] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:58.132448 15699 raft_consensus.cc:504] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.133128 15699 leader_election.cc:288] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:58.133641 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e8ea1f3484534ae16e968fab04860" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0" is_pre_election: true
I20240501 20:17:58.133664 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e8ea1f3484534ae16e968fab04860" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765" is_pre_election: true
I20240501 20:17:58.133845 14996 raft_consensus.cc:2454] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 0.
I20240501 20:17:58.133895 15258 raft_consensus.cc:2454] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 0.
I20240501 20:17:58.134300 15061 leader_election.cc:302] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e41b3aa39c7b4b0c90f3c36d6f07a0b4, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:58.134574 15699 raft_consensus.cc:2781] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:58.134725 15699 raft_consensus.cc:482] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:58.134814 15699 raft_consensus.cc:3037] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.136687 15699 raft_consensus.cc:504] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.137210 15699 leader_election.cc:288] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 election: Requested vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:58.137717 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e8ea1f3484534ae16e968fab04860" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765"
I20240501 20:17:58.137841 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e8ea1f3484534ae16e968fab04860" candidate_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0"
I20240501 20:17:58.137903 15258 raft_consensus.cc:3037] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.138034 14996 raft_consensus.cc:3037] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.140606 15258 raft_consensus.cc:2454] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 1.
I20240501 20:17:58.141084 15064 leader_election.cc:302] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, e41b3aa39c7b4b0c90f3c36d6f07a0b4; no voters:
I20240501 20:17:58.141428 15699 raft_consensus.cc:2781] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:58.141577 15699 raft_consensus.cc:686] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Becoming Leader. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Running, Role: LEADER
I20240501 20:17:58.141765 14996 raft_consensus.cc:2454] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41b3aa39c7b4b0c90f3c36d6f07a0b4 in term 1.
I20240501 20:17:58.141876 15699 consensus_queue.cc:237] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.145036 14855 catalog_manager.cc:5484] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: UNKNOWN } } }
I20240501 20:17:58.277096 15729 raft_consensus.cc:482] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:58.277308 15729 raft_consensus.cc:504] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.278043 15729 leader_election.cc:288] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097)
I20240501 20:17:58.278659 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d92d873304a308fc82d94081f3776" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0" is_pre_election: true
I20240501 20:17:58.278851 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d92d873304a308fc82d94081f3776" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" is_pre_election: true
I20240501 20:17:58.278970 14996 raft_consensus.cc:2454] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 0.
I20240501 20:17:58.279098 15127 raft_consensus.cc:2454] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 0.
I20240501 20:17:58.279445 15192 leader_election.cc:302] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:58.279750 15729 raft_consensus.cc:2781] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:58.279871 15729 raft_consensus.cc:482] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:58.279960 15729 raft_consensus.cc:3037] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.281971 15729 raft_consensus.cc:504] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.282564 15729 leader_election.cc:288] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 election: Requested vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097)
I20240501 20:17:58.283067 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d92d873304a308fc82d94081f3776" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee984883adca4b1fada1f685b5275ca0"
I20240501 20:17:58.283087 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d92d873304a308fc82d94081f3776" candidate_uuid: "0021804537544af482a273d050ee9765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
I20240501 20:17:58.283259 14996 raft_consensus.cc:3037] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.283274 15127 raft_consensus.cc:3037] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.285140 14996 raft_consensus.cc:2454] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 1.
I20240501 20:17:58.285195 15127 raft_consensus.cc:2454] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0021804537544af482a273d050ee9765 in term 1.
I20240501 20:17:58.285617 15192 leader_election.cc:302] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:58.285990 15729 raft_consensus.cc:2781] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:58.286161 15729 raft_consensus.cc:686] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Becoming Leader. State: Replica: 0021804537544af482a273d050ee9765, State: Running, Role: LEADER
I20240501 20:17:58.286485 15729 consensus_queue.cc:237] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.289386 14855 catalog_manager.cc:5484] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0021804537544af482a273d050ee9765 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } }
I20240501 20:17:58.324760 15611 raft_consensus.cc:482] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:17:58.324992 15611 raft_consensus.cc:504] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.325762 15611 leader_election.cc:288] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:58.326364 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925816a53f1e45a4a81efd7dfddd2b9d" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" is_pre_election: true
I20240501 20:17:58.326365 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925816a53f1e45a4a81efd7dfddd2b9d" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765" is_pre_election: true
I20240501 20:17:58.326687 15127 raft_consensus.cc:2454] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 0.
I20240501 20:17:58.326687 15258 raft_consensus.cc:2454] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 0.
I20240501 20:17:58.327199 14932 leader_election.cc:302] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:58.327492 15611 raft_consensus.cc:2781] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:17:58.327615 15611 raft_consensus.cc:482] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:17:58.327716 15611 raft_consensus.cc:3037] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.329665 15611 raft_consensus.cc:504] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.330232 15611 leader_election.cc:288] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 election: Requested vote from peers e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:17:58.330804 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925816a53f1e45a4a81efd7dfddd2b9d" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
I20240501 20:17:58.330801 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925816a53f1e45a4a81efd7dfddd2b9d" candidate_uuid: "ee984883adca4b1fada1f685b5275ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0021804537544af482a273d050ee9765"
I20240501 20:17:58.331033 15258 raft_consensus.cc:3037] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.331044 15127 raft_consensus.cc:3037] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:17:58.333873 15258 raft_consensus.cc:2454] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 1.
I20240501 20:17:58.333874 15127 raft_consensus.cc:2454] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee984883adca4b1fada1f685b5275ca0 in term 1.
I20240501 20:17:58.334375 14930 leader_election.cc:302] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, e41b3aa39c7b4b0c90f3c36d6f07a0b4, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:17:58.334745 15611 raft_consensus.cc:2781] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:17:58.334899 15611 raft_consensus.cc:686] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Becoming Leader. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Running, Role: LEADER
I20240501 20:17:58.335182 15611 consensus_queue.cc:237] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } }
I20240501 20:17:58.337884 14855 catalog_manager.cc:5484] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee984883adca4b1fada1f685b5275ca0 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: UNKNOWN } } }
I20240501 20:17:58.543902 15699 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:58.551414 15699 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:17:58.725556 15773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:58.725904 15773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:58.733707 15773 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 20:17:58.818902 15729 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:58.827744 15802 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:58.885670 15127 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:17:58.885881 15258 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:17:58.886587 15611 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:17:58.887291 15611 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:17:58.975150 14930 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:41097, user_credentials={real_user=slave}} blocked reactor thread for 88313.3us
I20240501 20:17:58.977054 15776 mvcc.cc:205] Tried to move back new op lower bound from 7022979804706816000 to 7022979802463125504. Current Snapshot: MvccSnapshot[applied={T|T < 7022979804706816000}]
I20240501 20:17:58.978147 15818 mvcc.cc:205] Tried to move back new op lower bound from 7022979804706816000 to 7022979802463125504. Current Snapshot: MvccSnapshot[applied={T|T < 7022979804706816000}]
INSERT report
rows total: 2000
time total: 264.927 ms
time per row: 0.132463 ms
I20240501 20:17:59.143647 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.197:41431
--local_ip_for_outbound_sockets=127.14.106.197
--tserver_master_addrs=127.14.106.254:39919
--webserver_port=40283
--webserver_interface=127.14.106.197
--builtin_ntp_servers=127.14.106.212:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:17:59.484853 15839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:59.485147 15839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:59.485298 15839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:59.492585 15839 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:17:59.492731 15839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:59.492894 15839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.197
I20240501 20:17:59.502223 15839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.197:41431
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.106.197
--webserver_port=40283
--tserver_master_addrs=127.14.106.254:39919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.197
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:59.503597 15839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:59.505406 15839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:59.517387 15844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:59.517906 15847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:59.518267 15845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:59.519131 15839 server_base.cc:1034] running on GCE node
I20240501 20:17:59.519775 15839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:59.520695 15839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:59.521904 15839 hybrid_clock.cc:648] HybridClock initialized: now 1714594679521857 us; error 41 us; skew 500 ppm
I20240501 20:17:59.522246 15839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:59.524562 15839 webserver.cc:460] Webserver started at http://127.14.106.197:40283/ using document root <none> and password file <none>
I20240501 20:17:59.525499 15839 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:59.525615 15839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:17:59.531060 15839 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 20:17:59.533757 15853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:59.535066 15839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:17:59.535205 15839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "79268a903afc4ef3a3bd79a32174fe2e"
format_stamp: "Formatted at 2024-05-01 20:17:51 on dist-test-slave-pg4x"
I20240501 20:17:59.535964 15839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:17:59.553689 15839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:17:59.554371 15839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:17:59.554613 15839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:17:59.555480 15839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:17:59.556890 15839 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:17:59.556962 15839 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:59.557055 15839 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:17:59.557098 15839 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:17:59.590809 15839 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.197:41431
I20240501 20:17:59.590881 15965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.197:41431 every 8 connection(s)
I20240501 20:17:59.592259 15839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:17:59.596861 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15839
I20240501 20:17:59.597625 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:35173
--local_ip_for_outbound_sockets=127.14.106.196
--tserver_master_addrs=127.14.106.254:39919
--webserver_port=36177
--webserver_interface=127.14.106.196
--builtin_ntp_servers=127.14.106.212:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:17:59.611420 15966 heartbeater.cc:347] Connected to a master server at 127.14.106.254:39919
I20240501 20:17:59.611768 15966 heartbeater.cc:464] Registering TS with master...
I20240501 20:17:59.612866 15966 heartbeater.cc:511] Master 127.14.106.254:39919 requested a full tablet report, sending...
I20240501 20:17:59.614712 14855 ts_manager.cc:194] Re-registered known tserver with Master: 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197:41431)
I20240501 20:17:59.615787 14855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.197:55871
W20240501 20:17:59.946828 15970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:17:59.947121 15970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:17:59.947268 15970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:17:59.954726 15970 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:17:59.954869 15970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:17:59.955037 15970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:17:59.966476 15970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:35173
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=36177
--tserver_master_addrs=127.14.106.254:39919
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:17:59.967811 15970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:17:59.969491 15970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:17:59.981272 15975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:59.981889 15976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:17:59.982542 15978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:17:59.983213 15970 server_base.cc:1034] running on GCE node
I20240501 20:17:59.983834 15970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:17:59.984798 15970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:17:59.986027 15970 hybrid_clock.cc:648] HybridClock initialized: now 1714594679985975 us; error 41 us; skew 500 ppm
I20240501 20:17:59.986356 15970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:17:59.993513 15970 webserver.cc:460] Webserver started at http://127.14.106.196:36177/ using document root <none> and password file <none>
I20240501 20:17:59.994397 15970 fs_manager.cc:359] Metadata directory not provided
I20240501 20:17:59.994553 15970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:18:00.000106 15970 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:18:00.003350 15984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:00.004732 15970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:18:00.004905 15970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "207d3f61d06541f5ae21e329b9fb5f8e"
format_stamp: "Formatted at 2024-05-01 20:17:50 on dist-test-slave-pg4x"
I20240501 20:18:00.005735 15970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:18:00.029793 15970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:18:00.030598 15970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:18:00.030895 15970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:18:00.031811 15970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:18:00.033277 15970 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:18:00.033350 15970 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:00.033445 15970 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:18:00.033489 15970 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:00.068749 15970 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:35173
I20240501 20:18:00.068784 16096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:35173 every 8 connection(s)
I20240501 20:18:00.070189 15970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:18:00.077925 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 15970
I20240501 20:18:00.082996 16097 heartbeater.cc:347] Connected to a master server at 127.14.106.254:39919
I20240501 20:18:00.083276 16097 heartbeater.cc:464] Registering TS with master...
I20240501 20:18:00.084065 16097 heartbeater.cc:511] Master 127.14.106.254:39919 requested a full tablet report, sending...
I20240501 20:18:00.085460 14848 ts_manager.cc:194] Re-registered known tserver with Master: 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196:35173)
I20240501 20:18:00.086323 14848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:51509
I20240501 20:18:00.583956 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:00.590014 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:00.618583 15966 heartbeater.cc:503] Master 127.14.106.254:39919 was elected leader, sending a full tablet report...
I20240501 20:18:00.632380 15107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:00.648986 15901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:00.650205 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:00.875479 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:00.902948 15901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:00.905301 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:00.921203 15107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:00.944928 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:01.089183 16097 heartbeater.cc:503] Master 127.14.106.254:39919 was elected leader, sending a full tablet report...
I20240501 20:18:01.115556 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:01.116884 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:01.130198 15901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:01.144914 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:01.146986 15107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:01.305544 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:01.306594 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:01.309648 15901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:01.340694 15107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:01.343575 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:01.402981 15127 consensus_queue.cc:237] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:01.405962 15259 raft_consensus.cc:1261] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.406098 14996 raft_consensus.cc:1261] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.406597 15699 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.406965 16241 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:18:01.416443 15061 consensus_peers.cc:485] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 -> Peer 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196:35173): Couldn't send request to peer 207d3f61d06541f5ae21e329b9fb5f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82e8ea1f3484534ae16e968fab04860. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:01.416275 15699 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.417181 14996 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.417394 15258 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.422044 14850 catalog_manager.cc:5484] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:01.439693 14996 consensus_queue.cc:237] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:01.442399 15127 raft_consensus.cc:1261] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.442679 15258 raft_consensus.cc:1261] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.443099 16256 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.443805 16257 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.446704 16256 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.447379 15127 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.448799 15258 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.452994 14849 catalog_manager.cc:5484] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
W20240501 20:18:01.457396 14931 consensus_peers.cc:485] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 -> Peer 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197:41431): Couldn't send request to peer 79268a903afc4ef3a3bd79a32174fe2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f062277a14ab4a6a83dbc7b1bc49eab4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:01.470572 15127 consensus_queue.cc:237] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:01.472981 14996 raft_consensus.cc:1261] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:18:01.473031 15061 consensus_peers.cc:485] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 -> Peer 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196:35173): Couldn't send request to peer 207d3f61d06541f5ae21e329b9fb5f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c134aed0124e4f35a57368d5855a0ff9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:01.473026 15258 raft_consensus.cc:1261] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.473559 16242 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.473858 15699 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.476703 15699 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.477430 14996 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.478963 15258 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.482277 14850 catalog_manager.cc:5484] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:01.496048 14996 consensus_queue.cc:237] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:01.498445 15127 raft_consensus.cc:1261] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:18:01.498577 14931 consensus_peers.cc:485] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 -> Peer 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197:41431): Couldn't send request to peer 79268a903afc4ef3a3bd79a32174fe2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23764dcc72b34239876036a7b99a6b6d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:01.498606 15258 raft_consensus.cc:1261] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.499050 16256 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.499688 16256 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.502085 15611 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.502899 15128 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.503545 15258 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.507503 14848 catalog_manager.cc:5484] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:01.520792 15258 consensus_queue.cc:237] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:01.523969 15128 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.524034 14996 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.524677 16283 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.525080 15834 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.535192 16288 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.539610 14995 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
W20240501 20:18:01.540241 15192 consensus_peers.cc:485] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 -> Peer 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196:35173): Couldn't send request to peer 207d3f61d06541f5ae21e329b9fb5f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 110b4b780c48418fbd1dc18c489c5fce. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:01.543327 14849 catalog_manager.cc:5484] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:01.545902 15128 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.557484 15258 consensus_queue.cc:237] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:01.559727 14995 raft_consensus.cc:1261] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.560359 16287 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.560523 15128 raft_consensus.cc:1261] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.561125 16287 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.564093 15834 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.565070 14995 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
W20240501 20:18:01.565963 15194 consensus_peers.cc:485] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 -> Peer 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197:41431): Couldn't send request to peer 79268a903afc4ef3a3bd79a32174fe2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 682d92d873304a308fc82d94081f3776. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:01.567356 15128 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.569767 14850 catalog_manager.cc:5484] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:01.594751 14995 consensus_queue.cc:237] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:01.598163 15128 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.598827 15611 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.599267 15258 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.600054 15611 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:18:01.604573 14929 consensus_peers.cc:485] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 -> Peer 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196:35173): Couldn't send request to peer 207d3f61d06541f5ae21e329b9fb5f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 925816a53f1e45a4a81efd7dfddd2b9d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:01.608403 16256 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.609133 15128 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.609452 15258 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.614202 14849 catalog_manager.cc:5484] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:01.635704 15258 consensus_queue.cc:237] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:01.638154 14995 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:01.638216 15128 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:18:01.638352 15192 consensus_peers.cc:485] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 -> Peer 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196:35173): Couldn't send request to peer 207d3f61d06541f5ae21e329b9fb5f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c32bb5190ebc49b5ad35a025746fa85d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:01.638896 16288 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.639219 16287 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:01.642740 16287 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.644217 15128 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.646816 14995 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } }
I20240501 20:18:01.648306 14849 catalog_manager.cc:5484] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index -1 to 4, NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:01.894213 16353 ts_tablet_manager.cc:902] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: Initiating tablet copy from peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097)
I20240501 20:18:01.895956 16353 tablet_copy_client.cc:308] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:41097
I20240501 20:18:01.902531 15148 tablet_copy_service.cc:140] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Received BeginTabletCopySession request for tablet b82e8ea1f3484534ae16e968fab04860 from peer 207d3f61d06541f5ae21e329b9fb5f8e ({username='slave'} at 127.14.106.196:57977)
I20240501 20:18:01.902794 15148 tablet_copy_service.cc:161] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Beginning new tablet copy session on tablet b82e8ea1f3484534ae16e968fab04860 from peer 207d3f61d06541f5ae21e329b9fb5f8e at {username='slave'} at 127.14.106.196:57977: session id = 207d3f61d06541f5ae21e329b9fb5f8e-b82e8ea1f3484534ae16e968fab04860
I20240501 20:18:01.905315 15148 tablet_copy_source_session.cc:205] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:01.909765 16353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82e8ea1f3484534ae16e968fab04860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:01.919917 16353 tablet_copy_client.cc:783] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:01.920188 16353 tablet_copy_client.cc:647] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:01.922554 16353 tablet_copy_client.cc:523] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:01.925769 16353 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap starting.
I20240501 20:18:01.938454 16353 log.cc:824] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: Log is configured to *not* fsync() on all Append() calls
I20240501 20:18:01.991317 16353 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:01.991815 16353 tablet_bootstrap.cc:492] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap complete.
I20240501 20:18:01.992251 16353 ts_tablet_manager.cc:1367] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent bootstrapping tablet: real 0.067s user 0.067s sys 0.000s
I20240501 20:18:01.997506 16353 raft_consensus.cc:348] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:01.997961 16353 raft_consensus.cc:729] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 207d3f61d06541f5ae21e329b9fb5f8e, State: Initialized, Role: LEARNER
I20240501 20:18:01.998522 16353 consensus_queue.cc:260] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.000142 16353 ts_tablet_manager.cc:1398] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 20:18:02.001853 15148 tablet_copy_service.cc:342] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Request end of tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-b82e8ea1f3484534ae16e968fab04860 received from {username='slave'} at 127.14.106.196:57977
I20240501 20:18:02.002060 15148 tablet_copy_service.cc:433] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: ending tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-b82e8ea1f3484534ae16e968fab04860 on tablet b82e8ea1f3484534ae16e968fab04860 with peer 207d3f61d06541f5ae21e329b9fb5f8e
I20240501 20:18:02.011380 16389 ts_tablet_manager.cc:902] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: Initiating tablet copy from peer ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715)
I20240501 20:18:02.014129 16389 tablet_copy_client.cc:308] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46715
I20240501 20:18:02.022414 15016 tablet_copy_service.cc:140] P ee984883adca4b1fada1f685b5275ca0: Received BeginTabletCopySession request for tablet 23764dcc72b34239876036a7b99a6b6d from peer 79268a903afc4ef3a3bd79a32174fe2e ({username='slave'} at 127.14.106.197:56007)
I20240501 20:18:02.022760 15016 tablet_copy_service.cc:161] P ee984883adca4b1fada1f685b5275ca0: Beginning new tablet copy session on tablet 23764dcc72b34239876036a7b99a6b6d from peer 79268a903afc4ef3a3bd79a32174fe2e at {username='slave'} at 127.14.106.197:56007: session id = 79268a903afc4ef3a3bd79a32174fe2e-23764dcc72b34239876036a7b99a6b6d
I20240501 20:18:02.025703 15016 tablet_copy_source_session.cc:205] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:02.031193 16389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23764dcc72b34239876036a7b99a6b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:02.036332 16408 ts_tablet_manager.cc:902] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: Initiating tablet copy from peer 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:18:02.043537 16408 tablet_copy_client.cc:308] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:45129
I20240501 20:18:02.046953 16389 tablet_copy_client.cc:783] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:02.047287 16389 tablet_copy_client.cc:647] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:02.049652 15279 tablet_copy_service.cc:140] P 0021804537544af482a273d050ee9765: Received BeginTabletCopySession request for tablet 682d92d873304a308fc82d94081f3776 from peer 79268a903afc4ef3a3bd79a32174fe2e ({username='slave'} at 127.14.106.197:48445)
I20240501 20:18:02.049878 15279 tablet_copy_service.cc:161] P 0021804537544af482a273d050ee9765: Beginning new tablet copy session on tablet 682d92d873304a308fc82d94081f3776 from peer 79268a903afc4ef3a3bd79a32174fe2e at {username='slave'} at 127.14.106.197:48445: session id = 79268a903afc4ef3a3bd79a32174fe2e-682d92d873304a308fc82d94081f3776
I20240501 20:18:02.050441 16389 tablet_copy_client.cc:523] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:02.052100 16413 ts_tablet_manager.cc:902] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: Initiating tablet copy from peer ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715)
I20240501 20:18:02.054306 15279 tablet_copy_source_session.cc:205] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:02.054487 16389 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap starting.
I20240501 20:18:02.056680 16408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682d92d873304a308fc82d94081f3776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:02.065101 16413 tablet_copy_client.cc:308] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46715
I20240501 20:18:02.065999 15016 tablet_copy_service.cc:140] P ee984883adca4b1fada1f685b5275ca0: Received BeginTabletCopySession request for tablet f062277a14ab4a6a83dbc7b1bc49eab4 from peer 79268a903afc4ef3a3bd79a32174fe2e ({username='slave'} at 127.14.106.197:56007)
I20240501 20:18:02.066191 15016 tablet_copy_service.cc:161] P ee984883adca4b1fada1f685b5275ca0: Beginning new tablet copy session on tablet f062277a14ab4a6a83dbc7b1bc49eab4 from peer 79268a903afc4ef3a3bd79a32174fe2e at {username='slave'} at 127.14.106.197:56007: session id = 79268a903afc4ef3a3bd79a32174fe2e-f062277a14ab4a6a83dbc7b1bc49eab4
I20240501 20:18:02.066268 16408 tablet_copy_client.cc:783] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:02.066553 16408 tablet_copy_client.cc:647] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:02.066576 16389 log.cc:824] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: Log is configured to *not* fsync() on all Append() calls
I20240501 20:18:02.067759 16353 ts_tablet_manager.cc:902] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: Initiating tablet copy from peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097)
I20240501 20:18:02.068962 15016 tablet_copy_source_session.cc:205] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:02.071453 16408 tablet_copy_client.cc:523] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:02.074378 16413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f062277a14ab4a6a83dbc7b1bc49eab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:02.075105 16408 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap starting.
I20240501 20:18:02.094992 16413 tablet_copy_client.cc:783] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:02.095338 16413 tablet_copy_client.cc:647] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:02.097286 16413 tablet_copy_client.cc:523] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:02.101153 16413 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap starting.
I20240501 20:18:02.120523 16353 tablet_copy_client.cc:308] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:41097
I20240501 20:18:02.121584 15148 tablet_copy_service.cc:140] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Received BeginTabletCopySession request for tablet c134aed0124e4f35a57368d5855a0ff9 from peer 207d3f61d06541f5ae21e329b9fb5f8e ({username='slave'} at 127.14.106.196:57977)
I20240501 20:18:02.121779 15148 tablet_copy_service.cc:161] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Beginning new tablet copy session on tablet c134aed0124e4f35a57368d5855a0ff9 from peer 207d3f61d06541f5ae21e329b9fb5f8e at {username='slave'} at 127.14.106.196:57977: session id = 207d3f61d06541f5ae21e329b9fb5f8e-c134aed0124e4f35a57368d5855a0ff9
I20240501 20:18:02.123919 16389 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:02.124614 16389 tablet_bootstrap.cc:492] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap complete.
I20240501 20:18:02.125185 16389 ts_tablet_manager.cc:1367] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: Time spent bootstrapping tablet: real 0.071s user 0.069s sys 0.000s
I20240501 20:18:02.125802 15148 tablet_copy_source_session.cc:205] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:02.132206 16389 raft_consensus.cc:348] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:02.133163 16426 ts_tablet_manager.cc:902] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: Initiating tablet copy from peer 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:18:02.134516 16427 ts_tablet_manager.cc:902] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: Initiating tablet copy from peer ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715)
I20240501 20:18:02.134836 16389 raft_consensus.cc:729] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79268a903afc4ef3a3bd79a32174fe2e, State: Initialized, Role: LEARNER
I20240501 20:18:02.136260 16389 consensus_queue.cc:260] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:02.137888 16408 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:02.138950 16353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c134aed0124e4f35a57368d5855a0ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:02.141690 16389 ts_tablet_manager.cc:1398] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20240501 20:18:02.152944 16408 tablet_bootstrap.cc:492] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap complete.
I20240501 20:18:02.153419 16408 ts_tablet_manager.cc:1367] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: Time spent bootstrapping tablet: real 0.078s user 0.042s sys 0.016s
I20240501 20:18:02.142132 16425 ts_tablet_manager.cc:902] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: Initiating tablet copy from peer 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:18:02.154464 16408 raft_consensus.cc:348] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:02.154773 16408 raft_consensus.cc:729] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79268a903afc4ef3a3bd79a32174fe2e, State: Initialized, Role: LEARNER
I20240501 20:18:02.155014 16408 consensus_queue.cc:260] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:02.155970 15016 tablet_copy_service.cc:342] P ee984883adca4b1fada1f685b5275ca0: Request end of tablet copy session 79268a903afc4ef3a3bd79a32174fe2e-23764dcc72b34239876036a7b99a6b6d received from {username='slave'} at 127.14.106.197:56007
I20240501 20:18:02.156131 15016 tablet_copy_service.cc:433] P ee984883adca4b1fada1f685b5275ca0: ending tablet copy session 79268a903afc4ef3a3bd79a32174fe2e-23764dcc72b34239876036a7b99a6b6d on tablet 23764dcc72b34239876036a7b99a6b6d with peer 79268a903afc4ef3a3bd79a32174fe2e
I20240501 20:18:02.161181 16353 tablet_copy_client.cc:783] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:02.161445 16353 tablet_copy_client.cc:647] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:02.163625 16408 ts_tablet_manager.cc:1398] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:18:02.164515 16353 tablet_copy_client.cc:523] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:02.165866 15279 tablet_copy_service.cc:342] P 0021804537544af482a273d050ee9765: Request end of tablet copy session 79268a903afc4ef3a3bd79a32174fe2e-682d92d873304a308fc82d94081f3776 received from {username='slave'} at 127.14.106.197:48445
I20240501 20:18:02.166059 15279 tablet_copy_service.cc:433] P 0021804537544af482a273d050ee9765: ending tablet copy session 79268a903afc4ef3a3bd79a32174fe2e-682d92d873304a308fc82d94081f3776 on tablet 682d92d873304a308fc82d94081f3776 with peer 79268a903afc4ef3a3bd79a32174fe2e
I20240501 20:18:02.168982 16427 tablet_copy_client.cc:308] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46715
I20240501 20:18:02.176200 16413 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:02.176707 16413 tablet_bootstrap.cc:492] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap complete.
I20240501 20:18:02.176827 15016 tablet_copy_service.cc:140] P ee984883adca4b1fada1f685b5275ca0: Received BeginTabletCopySession request for tablet 925816a53f1e45a4a81efd7dfddd2b9d from peer 207d3f61d06541f5ae21e329b9fb5f8e ({username='slave'} at 127.14.106.196:39849)
I20240501 20:18:02.176995 15016 tablet_copy_service.cc:161] P ee984883adca4b1fada1f685b5275ca0: Beginning new tablet copy session on tablet 925816a53f1e45a4a81efd7dfddd2b9d from peer 207d3f61d06541f5ae21e329b9fb5f8e at {username='slave'} at 127.14.106.196:39849: session id = 207d3f61d06541f5ae21e329b9fb5f8e-925816a53f1e45a4a81efd7dfddd2b9d
I20240501 20:18:02.177076 16413 ts_tablet_manager.cc:1367] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: Time spent bootstrapping tablet: real 0.076s user 0.060s sys 0.010s
I20240501 20:18:02.178154 16413 raft_consensus.cc:348] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:02.178512 16413 raft_consensus.cc:729] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79268a903afc4ef3a3bd79a32174fe2e, State: Initialized, Role: LEARNER
I20240501 20:18:02.178803 16413 consensus_queue.cc:260] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:02.179584 16413 ts_tablet_manager.cc:1398] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:18:02.180908 15016 tablet_copy_source_session.cc:205] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:02.181020 15015 tablet_copy_service.cc:342] P ee984883adca4b1fada1f685b5275ca0: Request end of tablet copy session 79268a903afc4ef3a3bd79a32174fe2e-f062277a14ab4a6a83dbc7b1bc49eab4 received from {username='slave'} at 127.14.106.197:56007
I20240501 20:18:02.181187 15015 tablet_copy_service.cc:433] P ee984883adca4b1fada1f685b5275ca0: ending tablet copy session 79268a903afc4ef3a3bd79a32174fe2e-f062277a14ab4a6a83dbc7b1bc49eab4 on tablet f062277a14ab4a6a83dbc7b1bc49eab4 with peer 79268a903afc4ef3a3bd79a32174fe2e
I20240501 20:18:02.182271 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925816a53f1e45a4a81efd7dfddd2b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:02.189045 16427 tablet_copy_client.cc:783] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:02.189311 16427 tablet_copy_client.cc:647] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:02.189510 16353 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap starting.
I20240501 20:18:02.191465 16427 tablet_copy_client.cc:523] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:02.192539 16426 tablet_copy_client.cc:308] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:45129
I20240501 20:18:02.196995 16427 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap starting.
I20240501 20:18:02.204432 15279 tablet_copy_service.cc:140] P 0021804537544af482a273d050ee9765: Received BeginTabletCopySession request for tablet c32bb5190ebc49b5ad35a025746fa85d from peer 207d3f61d06541f5ae21e329b9fb5f8e ({username='slave'} at 127.14.106.196:53461)
I20240501 20:18:02.204612 15279 tablet_copy_service.cc:161] P 0021804537544af482a273d050ee9765: Beginning new tablet copy session on tablet c32bb5190ebc49b5ad35a025746fa85d from peer 207d3f61d06541f5ae21e329b9fb5f8e at {username='slave'} at 127.14.106.196:53461: session id = 207d3f61d06541f5ae21e329b9fb5f8e-c32bb5190ebc49b5ad35a025746fa85d
I20240501 20:18:02.207585 15279 tablet_copy_source_session.cc:205] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:02.209048 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32bb5190ebc49b5ad35a025746fa85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:02.217213 16426 tablet_copy_client.cc:783] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:02.217501 16426 tablet_copy_client.cc:647] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:02.219658 16426 tablet_copy_client.cc:523] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:02.222563 16426 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap starting.
I20240501 20:18:02.223803 16425 tablet_copy_client.cc:308] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:45129
I20240501 20:18:02.224582 15279 tablet_copy_service.cc:140] P 0021804537544af482a273d050ee9765: Received BeginTabletCopySession request for tablet 110b4b780c48418fbd1dc18c489c5fce from peer 207d3f61d06541f5ae21e329b9fb5f8e ({username='slave'} at 127.14.106.196:53461)
I20240501 20:18:02.224749 15279 tablet_copy_service.cc:161] P 0021804537544af482a273d050ee9765: Beginning new tablet copy session on tablet 110b4b780c48418fbd1dc18c489c5fce from peer 207d3f61d06541f5ae21e329b9fb5f8e at {username='slave'} at 127.14.106.196:53461: session id = 207d3f61d06541f5ae21e329b9fb5f8e-110b4b780c48418fbd1dc18c489c5fce
I20240501 20:18:02.227784 15279 tablet_copy_source_session.cc:205] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:02.229110 16425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b4b780c48418fbd1dc18c489c5fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:02.235811 16425 tablet_copy_client.cc:783] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:02.236050 16425 tablet_copy_client.cc:647] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:02.237771 16425 tablet_copy_client.cc:523] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:02.248615 16425 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap starting.
I20240501 20:18:02.271687 16353 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:02.272240 16353 tablet_bootstrap.cc:492] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap complete.
I20240501 20:18:02.272639 16353 ts_tablet_manager.cc:1367] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent bootstrapping tablet: real 0.083s user 0.051s sys 0.012s
I20240501 20:18:02.273815 16353 raft_consensus.cc:348] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.274200 16427 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:02.274246 16353 raft_consensus.cc:729] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 207d3f61d06541f5ae21e329b9fb5f8e, State: Initialized, Role: LEARNER
I20240501 20:18:02.274578 16353 consensus_queue.cc:260] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.275144 16427 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap complete.
I20240501 20:18:02.275543 16427 ts_tablet_manager.cc:1367] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent bootstrapping tablet: real 0.079s user 0.055s sys 0.009s
I20240501 20:18:02.276644 16427 raft_consensus.cc:348] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.277053 16427 raft_consensus.cc:729] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 207d3f61d06541f5ae21e329b9fb5f8e, State: Initialized, Role: LEARNER
I20240501 20:18:02.277381 16427 consensus_queue.cc:260] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.278086 16427 ts_tablet_manager.cc:1398] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:18:02.279937 15016 tablet_copy_service.cc:342] P ee984883adca4b1fada1f685b5275ca0: Request end of tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-925816a53f1e45a4a81efd7dfddd2b9d received from {username='slave'} at 127.14.106.196:39849
I20240501 20:18:02.280074 15016 tablet_copy_service.cc:433] P ee984883adca4b1fada1f685b5275ca0: ending tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-925816a53f1e45a4a81efd7dfddd2b9d on tablet 925816a53f1e45a4a81efd7dfddd2b9d with peer 207d3f61d06541f5ae21e329b9fb5f8e
I20240501 20:18:02.283969 16353 ts_tablet_manager.cc:1398] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:18:02.286741 15148 tablet_copy_service.cc:342] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Request end of tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-c134aed0124e4f35a57368d5855a0ff9 received from {username='slave'} at 127.14.106.196:57977
I20240501 20:18:02.286913 15148 tablet_copy_service.cc:433] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: ending tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-c134aed0124e4f35a57368d5855a0ff9 on tablet c134aed0124e4f35a57368d5855a0ff9 with peer 207d3f61d06541f5ae21e329b9fb5f8e
I20240501 20:18:02.291154 16426 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:02.291559 16426 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap complete.
I20240501 20:18:02.291806 16426 ts_tablet_manager.cc:1367] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent bootstrapping tablet: real 0.069s user 0.060s sys 0.008s
I20240501 20:18:02.292658 16426 raft_consensus.cc:348] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.292937 16426 raft_consensus.cc:729] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 207d3f61d06541f5ae21e329b9fb5f8e, State: Initialized, Role: LEARNER
I20240501 20:18:02.293146 16426 consensus_queue.cc:260] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.293706 16426 ts_tablet_manager.cc:1398] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:18:02.294872 15279 tablet_copy_service.cc:342] P 0021804537544af482a273d050ee9765: Request end of tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-c32bb5190ebc49b5ad35a025746fa85d received from {username='slave'} at 127.14.106.196:53461
I20240501 20:18:02.295022 15279 tablet_copy_service.cc:433] P 0021804537544af482a273d050ee9765: ending tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-c32bb5190ebc49b5ad35a025746fa85d on tablet c32bb5190ebc49b5ad35a025746fa85d with peer 207d3f61d06541f5ae21e329b9fb5f8e
I20240501 20:18:02.311889 16425 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:02.312474 16425 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: Bootstrap complete.
I20240501 20:18:02.312769 16425 ts_tablet_manager.cc:1367] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent bootstrapping tablet: real 0.064s user 0.042s sys 0.007s
I20240501 20:18:02.313508 16425 raft_consensus.cc:348] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.313809 16425 raft_consensus.cc:729] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 207d3f61d06541f5ae21e329b9fb5f8e, State: Initialized, Role: LEARNER
I20240501 20:18:02.314011 16425 consensus_queue.cc:260] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } }
I20240501 20:18:02.314558 16425 ts_tablet_manager.cc:1398] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:18:02.315279 15279 tablet_copy_service.cc:342] P 0021804537544af482a273d050ee9765: Request end of tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-110b4b780c48418fbd1dc18c489c5fce received from {username='slave'} at 127.14.106.196:53461
I20240501 20:18:02.315443 15279 tablet_copy_service.cc:433] P 0021804537544af482a273d050ee9765: ending tablet copy session 207d3f61d06541f5ae21e329b9fb5f8e-110b4b780c48418fbd1dc18c489c5fce on tablet 110b4b780c48418fbd1dc18c489c5fce with peer 207d3f61d06541f5ae21e329b9fb5f8e
I20240501 20:18:02.358814 16052 raft_consensus.cc:1203] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:02.491130 15921 raft_consensus.cc:1203] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:02.498898 15921 raft_consensus.cc:1203] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:02.504491 16052 raft_consensus.cc:1203] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:02.545157 16052 raft_consensus.cc:1203] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:02.554656 16052 raft_consensus.cc:1203] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:02.624513 15921 raft_consensus.cc:1203] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:02.647511 16052 raft_consensus.cc:1203] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:02.918326 16480 raft_consensus.cc:1050] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: attempting to promote NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e to VOTER
I20240501 20:18:02.919847 16480 consensus_queue.cc:237] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:02.922691 16052 raft_consensus.cc:1261] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.923274 15258 raft_consensus.cc:1261] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.923686 14995 raft_consensus.cc:1261] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.923807 15699 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.924420 15699 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.925972 16480 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.929857 15699 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.930760 16052 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.934518 15258 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.935081 14995 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.941229 14855 catalog_manager.cc:5484] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.951249 15128 consensus_queue.cc:237] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:02.952919 16052 raft_consensus.cc:1261] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:02.953030 14995 raft_consensus.cc:1261] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:02.953451 16241 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:02.953821 15699 consensus_queue.cc:1035] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:02.955870 16480 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.956425 14995 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.959388 14840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b82e8ea1f3484534ae16e968fab04860 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:02.959219 16052 raft_consensus.cc:2932] T b82e8ea1f3484534ae16e968fab04860 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.960013 16257 raft_consensus.cc:1050] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: attempting to promote NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e to VOTER
I20240501 20:18:02.961061 16257 consensus_queue.cc:237] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:02.961607 14848 catalog_manager.cc:5484] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:02.964562 16487 raft_consensus.cc:1050] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765: attempting to promote NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e to VOTER
I20240501 20:18:02.965054 16050 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.965677 16487 consensus_queue.cc:237] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:02.965868 16489 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.967164 15128 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.965945 15258 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.967765 16489 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.968179 16488 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.970120 15921 raft_consensus.cc:1261] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.970223 15127 raft_consensus.cc:1261] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.970803 16288 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.972020 16486 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.973641 14995 raft_consensus.cc:1261] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.975883 16486 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:02.981717 16489 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.987546 15127 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.992329 15699 raft_consensus.cc:1050] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: attempting to promote NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e to VOTER
I20240501 20:18:02.992911 15258 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.993126 15699 consensus_queue.cc:237] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:02.994648 16050 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:02.997207 15238 tablet_service.cc:1514] Processing DeleteTablet for tablet b82e8ea1f3484534ae16e968fab04860 with delete_type TABLET_DATA_TOMBSTONED (TS 0021804537544af482a273d050ee9765 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45806
I20240501 20:18:02.997597 14849 catalog_manager.cc:5484] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:02.998128 16589 tablet_replica.cc:330] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: stopping tablet replica
I20240501 20:18:02.998847 15258 raft_consensus.cc:1261] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.999708 16052 raft_consensus.cc:1261] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:02.999778 14996 raft_consensus.cc:1261] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.000552 16241 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.001029 16480 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.001327 16241 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.003684 14994 consensus_queue.cc:237] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:03.005812 16050 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.006987 16257 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.010229 16257 raft_consensus.cc:1050] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0: attempting to promote NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e to VOTER
I20240501 20:18:03.011258 16257 consensus_queue.cc:237] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:03.012142 16488 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.015522 15128 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.018457 14839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 925816a53f1e45a4a81efd7dfddd2b9d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:03.018890 16489 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.020774 15128 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.022536 16589 raft_consensus.cc:2229] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:03.022997 16589 raft_consensus.cc:2258] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:03.020864 16050 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.026057 14849 catalog_manager.cc:5484] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:03.030045 16589 ts_tablet_manager.cc:1875] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:03.036597 15920 raft_consensus.cc:1261] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.037417 16581 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.037797 16583 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.039297 15259 raft_consensus.cc:1261] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.039633 15128 raft_consensus.cc:1261] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.040174 16581 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.041045 16581 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.031003 15238 tablet_service.cc:1514] Processing DeleteTablet for tablet 925816a53f1e45a4a81efd7dfddd2b9d with delete_type TABLET_DATA_TOMBSTONED (TS 0021804537544af482a273d050ee9765 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45806
I20240501 20:18:03.042506 15920 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.045545 14849 catalog_manager.cc:5484] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:03.046118 16589 ts_tablet_manager.cc:1888] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:03.046260 16589 log.cc:1196] T b82e8ea1f3484534ae16e968fab04860 P 0021804537544af482a273d050ee9765: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/b82e8ea1f3484534ae16e968fab04860
I20240501 20:18:03.047747 15127 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.051532 16589 tablet_replica.cc:330] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: stopping tablet replica
I20240501 20:18:03.051787 16589 raft_consensus.cc:2229] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:03.051762 14842 catalog_manager.cc:4830] TS 0021804537544af482a273d050ee9765 (127.14.106.195:45129): tablet b82e8ea1f3484534ae16e968fab04860 (table rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]) successfully deleted
I20240501 20:18:03.052218 16589 raft_consensus.cc:2258] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:03.052896 16480 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.053547 16486 raft_consensus.cc:1050] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: attempting to promote NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e to VOTER
I20240501 20:18:03.054419 16486 consensus_queue.cc:237] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:03.055017 16589 ts_tablet_manager.cc:1875] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:03.057765 16050 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.060547 15259 consensus_queue.cc:237] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:03.060725 15257 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.062546 15920 raft_consensus.cc:1261] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.062737 15127 raft_consensus.cc:1261] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.063246 16288 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.063716 16583 consensus_queue.cc:1035] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.066185 14995 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.066978 14848 catalog_manager.cc:5484] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.069944 16581 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.071605 15128 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.074957 15257 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.080452 16288 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.080206 15921 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.082420 15128 consensus_queue.cc:237] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:03.084019 15127 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.084290 15919 raft_consensus.cc:2932] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.085253 14850 catalog_manager.cc:5484] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.089102 14842 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 682d92d873304a308fc82d94081f3776 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:03.090526 16487 raft_consensus.cc:1050] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: attempting to promote NON_VOTER 207d3f61d06541f5ae21e329b9fb5f8e to VOTER
I20240501 20:18:03.091392 16487 consensus_queue.cc:237] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:03.094275 14848 catalog_manager.cc:5484] T 682d92d873304a308fc82d94081f3776 P 79268a903afc4ef3a3bd79a32174fe2e reported cstate change: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.096184 16589 ts_tablet_manager.cc:1888] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:03.096362 16589 log.cc:1196] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/925816a53f1e45a4a81efd7dfddd2b9d
I20240501 20:18:03.096230 14995 consensus_queue.cc:237] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:03.097146 14842 catalog_manager.cc:4830] TS 0021804537544af482a273d050ee9765 (127.14.106.195:45129): tablet 925816a53f1e45a4a81efd7dfddd2b9d (table rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]) successfully deleted
I20240501 20:18:03.097726 16488 raft_consensus.cc:1050] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0: attempting to promote NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e to VOTER
I20240501 20:18:03.098551 16488 consensus_queue.cc:237] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:03.098995 16050 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.099532 14996 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.100668 16486 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.100963 16583 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.101096 15127 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.103405 16050 raft_consensus.cc:1261] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.103891 16288 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.104164 15699 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.104967 15257 raft_consensus.cc:1261] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.105532 15699 consensus_queue.cc:1035] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.106806 16590 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.107375 16050 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.108608 15259 raft_consensus.cc:2932] T c134aed0124e4f35a57368d5855a0ff9 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.111215 14840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c134aed0124e4f35a57368d5855a0ff9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:03.112957 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet 682d92d873304a308fc82d94081f3776 with delete_type TABLET_DATA_TOMBSTONED (TS ee984883adca4b1fada1f685b5275ca0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36310
I20240501 20:18:03.113596 15920 raft_consensus.cc:1261] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.114226 16581 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.114459 14849 catalog_manager.cc:5484] T c134aed0124e4f35a57368d5855a0ff9 P 207d3f61d06541f5ae21e329b9fb5f8e reported cstate change: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.116688 14994 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.117513 16288 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.120107 14976 tablet_service.cc:1514] Processing DeleteTablet for tablet c134aed0124e4f35a57368d5855a0ff9 with delete_type TABLET_DATA_TOMBSTONED (TS ee984883adca4b1fada1f685b5275ca0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36310
I20240501 20:18:03.120815 15127 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.122043 16050 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 LEARNER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.121704 16489 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.126050 14839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f062277a14ab4a6a83dbc7b1bc49eab4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:03.127410 15919 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.132570 16583 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.132719 14849 catalog_manager.cc:5484] T f062277a14ab4a6a83dbc7b1bc49eab4 P 79268a903afc4ef3a3bd79a32174fe2e reported cstate change: config changed from index 5 to 6, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.133301 16583 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.137233 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet f062277a14ab4a6a83dbc7b1bc49eab4 with delete_type TABLET_DATA_TOMBSTONED (TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40254
I20240501 20:18:03.146173 15920 raft_consensus.cc:1261] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.146806 16581 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.149513 15259 raft_consensus.cc:1261] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.150048 16257 consensus_queue.cc:1035] T f062277a14ab4a6a83dbc7b1bc49eab4 P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.151495 15257 raft_consensus.cc:2932] T f062277a14ab4a6a83dbc7b1bc49eab4 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.157395 15126 raft_consensus.cc:1261] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.158118 16489 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.158965 15257 raft_consensus.cc:1261] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:03.160179 16489 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:03.163035 16604 tablet_replica.cc:330] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: stopping tablet replica
I20240501 20:18:03.163587 16604 raft_consensus.cc:2229] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:03.164014 16604 pending_rounds.cc:62] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: Trying to abort 1 pending ops.
I20240501 20:18:03.164125 16604 pending_rounds.cc:68] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022979821424807936 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "682d92d873304a308fc82d94081f3776" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } } }
I20240501 20:18:03.168044 16609 tablet_replica.cc:330] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: stopping tablet replica
I20240501 20:18:03.169180 16609 raft_consensus.cc:2229] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:03.169178 16583 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.169565 16486 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.174018 16609 raft_consensus.cc:2258] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:03.174309 15127 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.177007 14850 catalog_manager.cc:5484] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:03.177753 16052 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.186939 16604 raft_consensus.cc:2894] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:18:03.186889 14849 catalog_manager.cc:5484] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:03.187261 16604 raft_consensus.cc:2866] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:18:03.189168 16609 ts_tablet_manager.cc:1875] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:03.189460 15128 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.192011 15257 consensus_queue.cc:237] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:03.194988 15126 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.195644 16486 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.195255 16581 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.200850 16051 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.201440 15257 consensus_queue.cc:237] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } }
I20240501 20:18:03.197628 15127 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.208416 16049 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.211038 16609 ts_tablet_manager.cc:1888] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:03.211275 16609 log.cc:1196] T f062277a14ab4a6a83dbc7b1bc49eab4 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/f062277a14ab4a6a83dbc7b1bc49eab4
I20240501 20:18:03.211913 15259 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.213095 14995 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 207d3f61d06541f5ae21e329b9fb5f8e (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.213164 15919 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.215219 16486 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.216048 15125 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.216812 16604 raft_consensus.cc:2258] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:03.217236 16583 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.217654 14850 catalog_manager.cc:5484] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:03.219391 14842 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c32bb5190ebc49b5ad35a025746fa85d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:03.220408 16607 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.221486 16051 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.222187 16604 ts_tablet_manager.cc:1875] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:03.222684 14996 consensus_queue.cc:237] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:03.225131 14994 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.225893 14850 catalog_manager.cc:5484] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.227696 14855 catalog_manager.cc:5484] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e reported cstate change: config changed from index 5 to 6, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.228286 16487 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.228600 14842 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 110b4b780c48418fbd1dc18c489c5fce with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:03.229760 16050 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.229879 14840 catalog_manager.cc:4830] TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097): tablet f062277a14ab4a6a83dbc7b1bc49eab4 (table rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]) successfully deleted
I20240501 20:18:03.230553 16487 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.230540 14995 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.232174 16050 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } }
I20240501 20:18:03.233230 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b4b780c48418fbd1dc18c489c5fce with delete_type TABLET_DATA_TOMBSTONED (TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40254
I20240501 20:18:03.233928 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet c32bb5190ebc49b5ad35a025746fa85d with delete_type TABLET_DATA_TOMBSTONED (TS ee984883adca4b1fada1f685b5275ca0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36310
I20240501 20:18:03.235368 15920 raft_consensus.cc:1261] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.237241 16581 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.239440 16489 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.242973 16609 tablet_replica.cc:330] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: stopping tablet replica
I20240501 20:18:03.243268 14839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 23764dcc72b34239876036a7b99a6b6d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:03.243338 16609 raft_consensus.cc:2229] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:03.244244 15920 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.250759 14855 catalog_manager.cc:5484] T 23764dcc72b34239876036a7b99a6b6d P 79268a903afc4ef3a3bd79a32174fe2e reported cstate change: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.251667 16609 raft_consensus.cc:2258] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:03.252849 16609 ts_tablet_manager.cc:1875] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:03.254281 15125 raft_consensus.cc:1261] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:03.255231 16581 consensus_queue.cc:1035] T 23764dcc72b34239876036a7b99a6b6d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:03.257009 15238 tablet_service.cc:1514] Processing DeleteTablet for tablet 23764dcc72b34239876036a7b99a6b6d with delete_type TABLET_DATA_TOMBSTONED (TS 0021804537544af482a273d050ee9765 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45806
I20240501 20:18:03.257355 16589 tablet_replica.cc:330] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: stopping tablet replica
I20240501 20:18:03.257553 16589 raft_consensus.cc:2229] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:03.257282 15125 raft_consensus.cc:2932] T 23764dcc72b34239876036a7b99a6b6d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:03.257777 16589 raft_consensus.cc:2258] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:03.258843 16589 ts_tablet_manager.cc:1875] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:03.263532 16604 ts_tablet_manager.cc:1888] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:03.265229 16589 ts_tablet_manager.cc:1888] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:03.265448 16604 log.cc:1196] T 682d92d873304a308fc82d94081f3776 P ee984883adca4b1fada1f685b5275ca0: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/682d92d873304a308fc82d94081f3776
I20240501 20:18:03.265415 16589 log.cc:1196] T 23764dcc72b34239876036a7b99a6b6d P 0021804537544af482a273d050ee9765: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/23764dcc72b34239876036a7b99a6b6d
I20240501 20:18:03.266276 14842 catalog_manager.cc:4830] TS 0021804537544af482a273d050ee9765 (127.14.106.195:45129): tablet 23764dcc72b34239876036a7b99a6b6d (table rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]) successfully deleted
I20240501 20:18:03.266515 14839 catalog_manager.cc:4830] TS ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715): tablet 682d92d873304a308fc82d94081f3776 (table rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]) successfully deleted
I20240501 20:18:03.268229 16609 ts_tablet_manager.cc:1888] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:03.268353 16609 log.cc:1196] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/110b4b780c48418fbd1dc18c489c5fce
I20240501 20:18:03.269132 14840 catalog_manager.cc:4830] TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097): tablet 110b4b780c48418fbd1dc18c489c5fce (table rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]) successfully deleted
I20240501 20:18:03.266567 16604 tablet_replica.cc:330] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: stopping tablet replica
I20240501 20:18:03.270958 16604 raft_consensus.cc:2229] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:03.271307 16604 raft_consensus.cc:2258] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:03.272392 16604 ts_tablet_manager.cc:1875] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:03.278446 16604 ts_tablet_manager.cc:1888] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:03.278690 16604 log.cc:1196] T c134aed0124e4f35a57368d5855a0ff9 P ee984883adca4b1fada1f685b5275ca0: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/c134aed0124e4f35a57368d5855a0ff9
I20240501 20:18:03.279448 16604 tablet_replica.cc:330] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: stopping tablet replica
I20240501 20:18:03.279507 14839 catalog_manager.cc:4830] TS ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715): tablet c134aed0124e4f35a57368d5855a0ff9 (table rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]) successfully deleted
I20240501 20:18:03.279778 16604 raft_consensus.cc:2229] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:03.279982 16604 pending_rounds.cc:62] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Trying to abort 1 pending ops.
I20240501 20:18:03.280056 16604 pending_rounds.cc:68] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022979821940011008 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c32bb5190ebc49b5ad35a025746fa85d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } } }
I20240501 20:18:03.280402 16604 raft_consensus.cc:2894] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:18:03.280539 16604 raft_consensus.cc:2866] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:18:03.280653 16604 raft_consensus.cc:2258] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:03.281641 16604 ts_tablet_manager.cc:1875] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:03.287114 16604 ts_tablet_manager.cc:1888] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:03.287241 16604 log.cc:1196] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/c32bb5190ebc49b5ad35a025746fa85d
I20240501 20:18:03.287928 14839 catalog_manager.cc:4830] TS ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715): tablet c32bb5190ebc49b5ad35a025746fa85d (table rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]) successfully deleted
I20240501 20:18:03.423514 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:03.431077 15901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:03.434357 15107 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:03.448011 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:03.464233 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:03.520514 15125 consensus_queue.cc:237] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:03.523368 14995 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:03.523815 15259 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:03.524140 16599 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:03.525036 16599 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:03.527253 16590 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
W20240501 20:18:03.531013 15063 consensus_peers.cc:485] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 -> Peer 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197:41431): Couldn't send request to peer 79268a903afc4ef3a3bd79a32174fe2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3822c87b8cd44c4dbf432adf9aa0f590. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:03.531946 14995 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:03.533285 14850 catalog_manager.cc:5484] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:03.535291 15259 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } } }
I20240501 20:18:04.000191 16710 ts_tablet_manager.cc:902] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: Initiating tablet copy from peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097)
I20240501 20:18:04.001639 16710 tablet_copy_client.cc:308] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:41097
I20240501 20:18:04.007800 15148 tablet_copy_service.cc:140] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Received BeginTabletCopySession request for tablet 3822c87b8cd44c4dbf432adf9aa0f590 from peer 79268a903afc4ef3a3bd79a32174fe2e ({username='slave'} at 127.14.106.197:48213)
I20240501 20:18:04.007969 15148 tablet_copy_service.cc:161] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Beginning new tablet copy session on tablet 3822c87b8cd44c4dbf432adf9aa0f590 from peer 79268a903afc4ef3a3bd79a32174fe2e at {username='slave'} at 127.14.106.197:48213: session id = 79268a903afc4ef3a3bd79a32174fe2e-3822c87b8cd44c4dbf432adf9aa0f590
I20240501 20:18:04.010347 15148 tablet_copy_source_session.cc:205] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:04.011720 16710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3822c87b8cd44c4dbf432adf9aa0f590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:04.016091 16710 tablet_copy_client.cc:783] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:04.016345 16710 tablet_copy_client.cc:647] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:04.018080 16710 tablet_copy_client.cc:523] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:04.020372 16710 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap starting.
I20240501 20:18:04.050181 16710 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:04.050611 16710 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: Bootstrap complete.
I20240501 20:18:04.050925 16710 ts_tablet_manager.cc:1367] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.008s
I20240501 20:18:04.051565 16710 raft_consensus.cc:348] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:04.051784 16710 raft_consensus.cc:729] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79268a903afc4ef3a3bd79a32174fe2e, State: Initialized, Role: LEARNER
I20240501 20:18:04.051988 16710 consensus_queue.cc:260] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: NON_VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: true } }
I20240501 20:18:04.052696 16710 ts_tablet_manager.cc:1398] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:18:04.053509 15148 tablet_copy_service.cc:342] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Request end of tablet copy session 79268a903afc4ef3a3bd79a32174fe2e-3822c87b8cd44c4dbf432adf9aa0f590 received from {username='slave'} at 127.14.106.197:48213
I20240501 20:18:04.053648 15148 tablet_copy_service.cc:433] P e41b3aa39c7b4b0c90f3c36d6f07a0b4: ending tablet copy session 79268a903afc4ef3a3bd79a32174fe2e-3822c87b8cd44c4dbf432adf9aa0f590 on tablet 3822c87b8cd44c4dbf432adf9aa0f590 with peer 79268a903afc4ef3a3bd79a32174fe2e
I20240501 20:18:04.420866 15920 raft_consensus.cc:1203] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:04.910095 16728 raft_consensus.cc:1050] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: attempting to promote NON_VOTER 79268a903afc4ef3a3bd79a32174fe2e to VOTER
I20240501 20:18:04.911151 16728 consensus_queue.cc:237] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:04.913450 15259 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:04.913856 14995 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:04.914201 16729 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:04.914324 15920 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 LEARNER]: Refusing update from remote peer e41b3aa39c7b4b0c90f3c36d6f07a0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:04.914808 16729 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:04.915649 16599 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:04.918675 16729 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:04.920096 15920 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:04.920907 14995 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:04.922667 15259 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:04.923755 14850 catalog_manager.cc:5484] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 reported cstate change: config changed from index 4 to 5, 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:05.032958 15125 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590"
dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48812
I20240501 20:18:05.033210 15125 raft_consensus.cc:595] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:05.254004 15125 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590"
dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48842
I20240501 20:18:05.254215 15125 raft_consensus.cc:595] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:05.313072 16728 raft_consensus.cc:979] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: : Instructing follower 79268a903afc4ef3a3bd79a32174fe2e to start an election
I20240501 20:18:05.313251 16728 raft_consensus.cc:1067] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Signalling peer 79268a903afc4ef3a3bd79a32174fe2e to start an election
I20240501 20:18:05.314098 15920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590"
dest_uuid: "79268a903afc4ef3a3bd79a32174fe2e"
from {username='slave'} at 127.14.106.194:55859
I20240501 20:18:05.314354 15920 raft_consensus.cc:482] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:18:05.314567 15920 raft_consensus.cc:3037] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:05.316567 15920 raft_consensus.cc:504] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:05.318253 15920 leader_election.cc:288] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [CANDIDATE]: Term 2 election: Requested vote from peers ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715), e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097), 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:18:05.318586 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590" candidate_uuid: "79268a903afc4ef3a3bd79a32174fe2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ee984883adca4b1fada1f685b5275ca0"
I20240501 20:18:05.318665 15259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590" candidate_uuid: "79268a903afc4ef3a3bd79a32174fe2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0021804537544af482a273d050ee9765"
I20240501 20:18:05.318854 14995 raft_consensus.cc:3037] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:05.318900 15259 raft_consensus.cc:3037] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:05.319648 15125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3822c87b8cd44c4dbf432adf9aa0f590" candidate_uuid: "79268a903afc4ef3a3bd79a32174fe2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4"
I20240501 20:18:05.319842 15125 raft_consensus.cc:3032] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:18:05.319947 15125 raft_consensus.cc:729] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Running, Role: LEADER
I20240501 20:18:05.320253 15125 consensus_queue.cc:260] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:05.320721 15125 raft_consensus.cc:3037] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:05.321033 15259 raft_consensus.cc:2454] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79268a903afc4ef3a3bd79a32174fe2e in term 2.
I20240501 20:18:05.321667 14995 raft_consensus.cc:2454] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79268a903afc4ef3a3bd79a32174fe2e in term 2.
I20240501 20:18:05.322155 15854 leader_election.cc:302] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0021804537544af482a273d050ee9765, 79268a903afc4ef3a3bd79a32174fe2e, ee984883adca4b1fada1f685b5275ca0; no voters:
I20240501 20:18:05.322528 16745 raft_consensus.cc:2781] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:18:05.322948 16745 raft_consensus.cc:686] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 2 LEADER]: Becoming Leader. State: Replica: 79268a903afc4ef3a3bd79a32174fe2e, State: Running, Role: LEADER
I20240501 20:18:05.323136 15125 raft_consensus.cc:2454] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79268a903afc4ef3a3bd79a32174fe2e in term 2.
I20240501 20:18:05.323443 16745 consensus_queue.cc:237] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:05.326660 14848 catalog_manager.cc:5484] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e reported cstate change: term changed from 1 to 2, leader changed from e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) to 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197). New cstate: current_term: 2 leader_uuid: "79268a903afc4ef3a3bd79a32174fe2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:05.726998 14995 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:18:05.727672 16745 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:05.898288 15259 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:18:05.898876 16745 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:05.903692 15125 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:18:05.905845 16787 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:05.918035 15920 consensus_queue.cc:237] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } }
I20240501 20:18:05.919855 15259 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:18:05.919992 14995 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:18:05.920460 16791 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:05.920680 16787 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:05.922735 16791 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:05.923410 14995 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:05.924808 15259 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } }
I20240501 20:18:05.926610 14841 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3822c87b8cd44c4dbf432adf9aa0f590 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:05.928378 14849 catalog_manager.cc:5484] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e reported cstate change: config changed from index 5 to 7, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "79268a903afc4ef3a3bd79a32174fe2e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:05.932963 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet 3822c87b8cd44c4dbf432adf9aa0f590 with delete_type TABLET_DATA_TOMBSTONED (TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40254
I20240501 20:18:05.937619 16798 tablet_replica.cc:330] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: stopping tablet replica
I20240501 20:18:05.937811 16798 raft_consensus.cc:2229] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:05.938005 16798 raft_consensus.cc:2258] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:05.938807 16798 ts_tablet_manager.cc:1875] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:05.943189 16798 ts_tablet_manager.cc:1888] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:18:05.943300 16798 log.cc:1196] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/3822c87b8cd44c4dbf432adf9aa0f590
I20240501 20:18:05.944051 14840 catalog_manager.cc:4830] TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097): tablet 3822c87b8cd44c4dbf432adf9aa0f590 (table rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]) successfully deleted
I20240501 20:18:06.194528 15901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:06.199393 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:06.214612 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:06.218003 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:06.236622 15107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:06.387653 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:06.391738 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:06.401274 15107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:06.401876 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:06.405164 15901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:06.489488 15919 consensus_queue.cc:237] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } }
I20240501 20:18:06.491724 15259 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 20:18:06.492316 15855 consensus_peers.cc:485] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e -> Peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097): Couldn't send request to peer e41b3aa39c7b4b0c90f3c36d6f07a0b4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:06.492372 16881 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:06.492607 14995 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:18:06.493139 16880 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:06.494906 16881 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } } }
I20240501 20:18:06.495662 15259 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } } }
I20240501 20:18:06.495915 14995 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } } }
I20240501 20:18:06.501163 14849 catalog_manager.cc:5484] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e reported cstate change: config changed from index 7 to 8, NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) added. New cstate: current_term: 2 leader_uuid: "79268a903afc4ef3a3bd79a32174fe2e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:06.524464 15259 consensus_queue.cc:237] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } }
I20240501 20:18:06.526966 16052 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:06.527603 16487 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:06.527874 14995 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:18:06.527899 15193 consensus_peers.cc:485] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 -> Peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097): Couldn't send request to peer e41b3aa39c7b4b0c90f3c36d6f07a0b4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:06.528512 16487 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:06.530655 16892 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } } }
I20240501 20:18:06.531255 16052 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } } }
I20240501 20:18:06.532002 14995 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } } }
I20240501 20:18:06.537771 14849 catalog_manager.cc:5484] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e reported cstate change: config changed from index 6 to 7, NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) added. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } } }
I20240501 20:18:06.551316 14995 consensus_queue.cc:237] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: true } }
I20240501 20:18:06.553853 16052 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:06.554076 15125 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:06.554488 16903 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:06.554973 16581 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:18:06.555272 14932 consensus_peers.cc:485] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 -> Peer 0021804537544af482a273d050ee9765 (127.14.106.195:45129): Couldn't send request to peer 0021804537544af482a273d050ee9765. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:06.557643 16882 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: true } } }
I20240501 20:18:06.558274 16052 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: true } } }
I20240501 20:18:06.560773 15125 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: true } } }
I20240501 20:18:06.562072 14855 catalog_manager.cc:5484] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 6 to 7, NON_VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) added. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:06.576366 15259 consensus_queue.cc:237] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: true } }
I20240501 20:18:06.578891 15125 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:18:06.579238 15192 consensus_peers.cc:485] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 -> Peer ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715): Couldn't send request to peer ee984883adca4b1fada1f685b5275ca0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:06.579260 16052 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:06.579465 16892 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:06.579886 16896 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:06.582223 16892 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: true } } }
I20240501 20:18:06.583489 16052 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: true } } }
I20240501 20:18:06.583451 15125 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: true } } }
I20240501 20:18:06.588254 14848 catalog_manager.cc:5484] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index 6 to 7, NON_VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) added. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:06.928182 16955 ts_tablet_manager.cc:902] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Initiating tablet copy from peer 79268a903afc4ef3a3bd79a32174fe2e (127.14.106.197:41431)
I20240501 20:18:06.929910 16955 tablet_copy_client.cc:308] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.197:41431
I20240501 20:18:06.930994 15941 tablet_copy_service.cc:140] P 79268a903afc4ef3a3bd79a32174fe2e: Received BeginTabletCopySession request for tablet 3822c87b8cd44c4dbf432adf9aa0f590 from peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 ({username='slave'} at 127.14.106.194:55859)
I20240501 20:18:06.931226 15941 tablet_copy_service.cc:161] P 79268a903afc4ef3a3bd79a32174fe2e: Beginning new tablet copy session on tablet 3822c87b8cd44c4dbf432adf9aa0f590 from peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 at {username='slave'} at 127.14.106.194:55859: session id = e41b3aa39c7b4b0c90f3c36d6f07a0b4-3822c87b8cd44c4dbf432adf9aa0f590
I20240501 20:18:06.933936 15941 tablet_copy_source_session.cc:205] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:06.935632 16955 ts_tablet_manager.cc:1875] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:06.942456 16955 ts_tablet_manager.cc:1888] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 20:18:06.942821 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3822c87b8cd44c4dbf432adf9aa0f590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:06.945659 16955 tablet_copy_client.cc:783] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:06.945936 16955 tablet_copy_client.cc:647] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:06.948040 16955 tablet_copy_client.cc:523] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:06.950928 16955 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:18:06.959388 16961 ts_tablet_manager.cc:902] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Initiating tablet copy from peer 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:18:06.961086 16961 tablet_copy_client.cc:308] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:45129
I20240501 20:18:06.961889 15279 tablet_copy_service.cc:140] P 0021804537544af482a273d050ee9765: Received BeginTabletCopySession request for tablet 110b4b780c48418fbd1dc18c489c5fce from peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 ({username='slave'} at 127.14.106.194:39839)
I20240501 20:18:06.962070 15279 tablet_copy_service.cc:161] P 0021804537544af482a273d050ee9765: Beginning new tablet copy session on tablet 110b4b780c48418fbd1dc18c489c5fce from peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 at {username='slave'} at 127.14.106.194:39839: session id = e41b3aa39c7b4b0c90f3c36d6f07a0b4-110b4b780c48418fbd1dc18c489c5fce
I20240501 20:18:06.964143 15279 tablet_copy_source_session.cc:205] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:06.965273 16961 ts_tablet_manager.cc:1875] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:06.970738 16961 ts_tablet_manager.cc:1888] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:06.971246 16961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b4b780c48418fbd1dc18c489c5fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:06.973342 16961 tablet_copy_client.cc:783] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:06.973600 16961 tablet_copy_client.cc:647] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:06.975425 16961 tablet_copy_client.cc:523] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:06.977967 16961 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap starting.
I20240501 20:18:06.987782 16955 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:06.988212 16955 tablet_bootstrap.cc:492] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap complete.
I20240501 20:18:06.988463 16955 ts_tablet_manager.cc:1367] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.038s user 0.023s sys 0.013s
I20240501 20:18:06.989126 16955 raft_consensus.cc:348] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } }
I20240501 20:18:06.989352 16955 raft_consensus.cc:729] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: LEARNER
I20240501 20:18:06.989569 16955 consensus_queue.cc:260] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } }
I20240501 20:18:06.990126 16955 ts_tablet_manager.cc:1398] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:18:06.992475 15941 tablet_copy_service.cc:342] P 79268a903afc4ef3a3bd79a32174fe2e: Request end of tablet copy session e41b3aa39c7b4b0c90f3c36d6f07a0b4-3822c87b8cd44c4dbf432adf9aa0f590 received from {username='slave'} at 127.14.106.194:55859
I20240501 20:18:06.992672 15941 tablet_copy_service.cc:433] P 79268a903afc4ef3a3bd79a32174fe2e: ending tablet copy session e41b3aa39c7b4b0c90f3c36d6f07a0b4-3822c87b8cd44c4dbf432adf9aa0f590 on tablet 3822c87b8cd44c4dbf432adf9aa0f590 with peer e41b3aa39c7b4b0c90f3c36d6f07a0b4
I20240501 20:18:07.013888 16961 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:07.014271 16961 tablet_bootstrap.cc:492] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Bootstrap complete.
I20240501 20:18:07.014520 16961 ts_tablet_manager.cc:1367] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent bootstrapping tablet: real 0.037s user 0.035s sys 0.000s
I20240501 20:18:07.015126 16961 raft_consensus.cc:348] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } }
I20240501 20:18:07.015337 16961 raft_consensus.cc:729] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e41b3aa39c7b4b0c90f3c36d6f07a0b4, State: Initialized, Role: LEARNER
I20240501 20:18:07.015539 16961 consensus_queue.cc:260] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: NON_VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: true } }
I20240501 20:18:07.016045 16961 ts_tablet_manager.cc:1398] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 20:18:07.016891 15279 tablet_copy_service.cc:342] P 0021804537544af482a273d050ee9765: Request end of tablet copy session e41b3aa39c7b4b0c90f3c36d6f07a0b4-110b4b780c48418fbd1dc18c489c5fce received from {username='slave'} at 127.14.106.194:39839
I20240501 20:18:07.017000 15279 tablet_copy_service.cc:433] P 0021804537544af482a273d050ee9765: ending tablet copy session e41b3aa39c7b4b0c90f3c36d6f07a0b4-110b4b780c48418fbd1dc18c489c5fce on tablet 110b4b780c48418fbd1dc18c489c5fce with peer e41b3aa39c7b4b0c90f3c36d6f07a0b4
I20240501 20:18:07.028808 16967 ts_tablet_manager.cc:902] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Initiating tablet copy from peer ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715)
I20240501 20:18:07.030423 16967 tablet_copy_client.cc:308] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:46715
I20240501 20:18:07.031267 15016 tablet_copy_service.cc:140] P ee984883adca4b1fada1f685b5275ca0: Received BeginTabletCopySession request for tablet 925816a53f1e45a4a81efd7dfddd2b9d from peer 0021804537544af482a273d050ee9765 ({username='slave'} at 127.14.106.195:55271)
I20240501 20:18:07.031447 15016 tablet_copy_service.cc:161] P ee984883adca4b1fada1f685b5275ca0: Beginning new tablet copy session on tablet 925816a53f1e45a4a81efd7dfddd2b9d from peer 0021804537544af482a273d050ee9765 at {username='slave'} at 127.14.106.195:55271: session id = 0021804537544af482a273d050ee9765-925816a53f1e45a4a81efd7dfddd2b9d
I20240501 20:18:07.033461 15016 tablet_copy_source_session.cc:205] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:07.034827 16967 ts_tablet_manager.cc:1875] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:07.039536 16967 ts_tablet_manager.cc:1888] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:07.039788 16967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925816a53f1e45a4a81efd7dfddd2b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:07.041967 16967 tablet_copy_client.cc:783] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:07.042222 16967 tablet_copy_client.cc:647] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:07.044251 16967 tablet_copy_client.cc:523] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:07.047014 16967 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Bootstrap starting.
I20240501 20:18:07.084292 16967 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:07.084869 16967 tablet_bootstrap.cc:492] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Bootstrap complete.
I20240501 20:18:07.085191 16967 ts_tablet_manager.cc:1367] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Time spent bootstrapping tablet: real 0.038s user 0.037s sys 0.000s
I20240501 20:18:07.086846 16967 raft_consensus.cc:348] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: true } }
I20240501 20:18:07.087122 16967 raft_consensus.cc:729] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0021804537544af482a273d050ee9765, State: Initialized, Role: LEARNER
I20240501 20:18:07.087370 16967 consensus_queue.cc:260] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: true } }
I20240501 20:18:07.088091 16967 ts_tablet_manager.cc:1398] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:18:07.089215 15016 tablet_copy_service.cc:342] P ee984883adca4b1fada1f685b5275ca0: Request end of tablet copy session 0021804537544af482a273d050ee9765-925816a53f1e45a4a81efd7dfddd2b9d received from {username='slave'} at 127.14.106.195:55271
I20240501 20:18:07.089375 15016 tablet_copy_service.cc:433] P ee984883adca4b1fada1f685b5275ca0: ending tablet copy session 0021804537544af482a273d050ee9765-925816a53f1e45a4a81efd7dfddd2b9d on tablet 925816a53f1e45a4a81efd7dfddd2b9d with peer 0021804537544af482a273d050ee9765
I20240501 20:18:07.188896 16993 ts_tablet_manager.cc:902] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Initiating tablet copy from peer 0021804537544af482a273d050ee9765 (127.14.106.195:45129)
I20240501 20:18:07.190536 16993 tablet_copy_client.cc:308] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:45129
I20240501 20:18:07.191480 15279 tablet_copy_service.cc:140] P 0021804537544af482a273d050ee9765: Received BeginTabletCopySession request for tablet c32bb5190ebc49b5ad35a025746fa85d from peer ee984883adca4b1fada1f685b5275ca0 ({username='slave'} at 127.14.106.193:49917)
I20240501 20:18:07.191629 15279 tablet_copy_service.cc:161] P 0021804537544af482a273d050ee9765: Beginning new tablet copy session on tablet c32bb5190ebc49b5ad35a025746fa85d from peer ee984883adca4b1fada1f685b5275ca0 at {username='slave'} at 127.14.106.193:49917: session id = ee984883adca4b1fada1f685b5275ca0-c32bb5190ebc49b5ad35a025746fa85d
I20240501 20:18:07.193619 15279 tablet_copy_source_session.cc:205] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:07.195072 16993 ts_tablet_manager.cc:1875] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:07.199798 16993 ts_tablet_manager.cc:1888] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:07.200102 16993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32bb5190ebc49b5ad35a025746fa85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:07.202564 16993 tablet_copy_client.cc:783] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:07.202852 16993 tablet_copy_client.cc:647] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:07.204708 16993 tablet_copy_client.cc:523] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:07.207499 16993 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Bootstrap starting.
I20240501 20:18:07.241526 16993 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:07.241981 16993 tablet_bootstrap.cc:492] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Bootstrap complete.
I20240501 20:18:07.242251 16993 ts_tablet_manager.cc:1367] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.007s
I20240501 20:18:07.242987 16993 raft_consensus.cc:348] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: true } }
I20240501 20:18:07.243219 16993 raft_consensus.cc:729] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee984883adca4b1fada1f685b5275ca0, State: Initialized, Role: LEARNER
I20240501 20:18:07.243419 16993 consensus_queue.cc:260] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: true } }
I20240501 20:18:07.244004 16993 ts_tablet_manager.cc:1398] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:18:07.244979 15279 tablet_copy_service.cc:342] P 0021804537544af482a273d050ee9765: Request end of tablet copy session ee984883adca4b1fada1f685b5275ca0-c32bb5190ebc49b5ad35a025746fa85d received from {username='slave'} at 127.14.106.193:49917
I20240501 20:18:07.246069 15279 tablet_copy_service.cc:433] P 0021804537544af482a273d050ee9765: ending tablet copy session ee984883adca4b1fada1f685b5275ca0-c32bb5190ebc49b5ad35a025746fa85d on tablet c32bb5190ebc49b5ad35a025746fa85d with peer ee984883adca4b1fada1f685b5275ca0
I20240501 20:18:07.449332 15125 raft_consensus.cc:1203] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:18:07.494881 15125 raft_consensus.cc:1203] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:18:07.505903 15259 raft_consensus.cc:1203] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:18:07.606395 14995 raft_consensus.cc:1203] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:18:08.031039 17066 raft_consensus.cc:1050] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e: attempting to promote NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 to VOTER
I20240501 20:18:08.031834 17066 consensus_queue.cc:237] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } }
I20240501 20:18:08.034274 15125 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 LEARNER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:18:08.034615 15257 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:18:08.034948 14995 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:18:08.035056 17068 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:08.035523 17021 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:08.035889 17066 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:08.039606 17066 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.040611 14995 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.041389 15257 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.046631 14849 catalog_manager.cc:5484] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 8 to 9, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "79268a903afc4ef3a3bd79a32174fe2e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.047820 15125 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { replace: true } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.060525 17078 raft_consensus.cc:1050] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0: attempting to promote NON_VOTER 0021804537544af482a273d050ee9765 to VOTER
I20240501 20:18:08.061645 17078 consensus_queue.cc:237] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } }
I20240501 20:18:08.067445 17045 raft_consensus.cc:1050] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765: attempting to promote NON_VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 to VOTER
I20240501 20:18:08.068392 17045 consensus_queue.cc:237] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } }
I20240501 20:18:08.070515 16052 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.070748 15257 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 LEARNER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.071631 15125 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.072331 17024 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:08.072949 17024 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:08.073390 17024 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:08.074986 15919 consensus_queue.cc:237] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } }
I20240501 20:18:08.078855 14995 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:08.079536 17066 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:08.080598 15125 raft_consensus.cc:1261] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 FOLLOWER]: Refusing update from remote peer 79268a903afc4ef3a3bd79a32174fe2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:08.081643 17066 consensus_queue.cc:1035] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:08.084259 17066 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.087719 15128 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.087715 14995 raft_consensus.cc:2932] T 3822c87b8cd44c4dbf432adf9aa0f590 P ee984883adca4b1fada1f685b5275ca0 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.089377 15125 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 LEARNER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.089579 14841 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3822c87b8cd44c4dbf432adf9aa0f590 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:08.090070 16896 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:08.091522 14850 catalog_manager.cc:5484] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e reported cstate change: config changed from index 9 to 10, VOTER 0021804537544af482a273d050ee9765 (127.14.106.195) evicted. New cstate: current_term: 2 leader_uuid: "79268a903afc4ef3a3bd79a32174fe2e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79268a903afc4ef3a3bd79a32174fe2e" member_type: VOTER last_known_addr { host: "127.14.106.197" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:08.094535 17024 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0021804537544af482a273d050ee9765 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } } }
I20240501 20:18:08.097990 16896 raft_consensus.cc:1050] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765: attempting to promote NON_VOTER ee984883adca4b1fada1f685b5275ca0 to VOTER
I20240501 20:18:08.099113 16896 consensus_queue.cc:237] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } }
I20240501 20:18:08.097643 16052 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0021804537544af482a273d050ee9765 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } } }
I20240501 20:18:08.104897 15257 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0021804537544af482a273d050ee9765 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } } }
I20240501 20:18:08.107570 15128 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0021804537544af482a273d050ee9765 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } } }
I20240501 20:18:08.109664 14995 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.110796 17045 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:08.111135 16050 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.112324 14849 catalog_manager.cc:5484] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index 7 to 8, 0021804537544af482a273d050ee9765 (127.14.106.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } } }
I20240501 20:18:08.114995 17045 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:08.121598 15238 tablet_service.cc:1514] Processing DeleteTablet for tablet 3822c87b8cd44c4dbf432adf9aa0f590 with delete_type TABLET_DATA_TOMBSTONED (TS 0021804537544af482a273d050ee9765 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45806
I20240501 20:18:08.134887 17022 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.135772 14994 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.136047 17094 tablet_replica.cc:330] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: stopping tablet replica
I20240501 20:18:08.136747 17094 raft_consensus.cc:2229] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:08.137387 17094 raft_consensus.cc:2258] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:08.138859 17094 ts_tablet_manager.cc:1875] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:08.142617 14995 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEARNER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.149284 17045 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20240501 20:18:08.151954 16050 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.153409 16050 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.156102 17098 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:08.164965 17094 ts_tablet_manager.cc:1888] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:18:08.166368 17094 log.cc:1196] T 3822c87b8cd44c4dbf432adf9aa0f590 P 0021804537544af482a273d050ee9765: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/3822c87b8cd44c4dbf432adf9aa0f590
I20240501 20:18:08.173355 17098 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } } }
I20240501 20:18:08.174391 14995 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } } }
I20240501 20:18:08.181227 14855 catalog_manager.cc:5484] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 7 to 8, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.184441 14994 consensus_queue.cc:237] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } }
I20240501 20:18:08.186820 15257 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:08.187741 17085 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:18:08.189340 15193 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.194:41097, user_credentials={real_user=slave}} blocked reactor thread for 44194.8us
I20240501 20:18:08.190265 15127 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:08.191040 17024 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } } }
I20240501 20:18:08.191736 14842 catalog_manager.cc:4830] TS 0021804537544af482a273d050ee9765 (127.14.106.195:45129): tablet 3822c87b8cd44c4dbf432adf9aa0f590 (table rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]) successfully deleted
I20240501 20:18:08.191512 15127 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.191591 15257 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 0021804537544af482a273d050ee9765 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } } }
I20240501 20:18:08.196400 17097 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:08.197502 14839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 925816a53f1e45a4a81efd7dfddd2b9d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:08.198151 14855 catalog_manager.cc:5484] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 8 to 9, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "ee984883adca4b1fada1f685b5275ca0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:08.202672 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet 925816a53f1e45a4a81efd7dfddd2b9d with delete_type TABLET_DATA_TOMBSTONED (TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40254
I20240501 20:18:08.202642 15257 consensus_queue.cc:237] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } }
I20240501 20:18:08.207407 16050 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:08.208185 17097 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:08.209491 15128 raft_consensus.cc:1261] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:08.210224 17023 consensus_queue.cc:1035] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:08.211637 16896 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.212286 16050 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.214215 15128 raft_consensus.cc:2932] T 110b4b780c48418fbd1dc18c489c5fce P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.217821 16051 raft_consensus.cc:1261] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer ee984883adca4b1fada1f685b5275ca0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:08.218587 17078 consensus_queue.cc:1035] T 925816a53f1e45a4a81efd7dfddd2b9d P ee984883adca4b1fada1f685b5275ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:08.220314 16051 raft_consensus.cc:2932] T 925816a53f1e45a4a81efd7dfddd2b9d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } attrs { promote: false } } }
I20240501 20:18:08.222586 14848 catalog_manager.cc:5484] T 110b4b780c48418fbd1dc18c489c5fce P 207d3f61d06541f5ae21e329b9fb5f8e reported cstate change: config changed from index 8 to 9, VOTER ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { promote: false } } }
I20240501 20:18:08.222323 14855 catalog_manager.cc:5484] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 reported cstate change: config changed from index 7 to 8, ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } } }
I20240501 20:18:08.226255 17102 tablet_replica.cc:330] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: stopping tablet replica
I20240501 20:18:08.226469 17102 raft_consensus.cc:2229] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:08.227473 14842 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 110b4b780c48418fbd1dc18c489c5fce with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:08.227821 17102 raft_consensus.cc:2258] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:08.228484 15127 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } } }
I20240501 20:18:08.229197 17102 ts_tablet_manager.cc:1875] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:08.230051 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b4b780c48418fbd1dc18c489c5fce with delete_type TABLET_DATA_TOMBSTONED (TS ee984883adca4b1fada1f685b5275ca0 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36310
I20240501 20:18:08.230859 17105 tablet_replica.cc:330] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: stopping tablet replica
I20240501 20:18:08.231113 17105 raft_consensus.cc:2229] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:08.231426 17105 raft_consensus.cc:2258] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:08.231912 16051 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee984883adca4b1fada1f685b5275ca0 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e41b3aa39c7b4b0c90f3c36d6f07a0b4" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 41097 } attrs { replace: true } } peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } } }
I20240501 20:18:08.233404 17105 ts_tablet_manager.cc:1875] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:08.237172 15257 consensus_queue.cc:237] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } }
I20240501 20:18:08.239553 14995 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:08.239857 17105 ts_tablet_manager.cc:1888] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:18:08.240027 17105 log.cc:1196] T 110b4b780c48418fbd1dc18c489c5fce P ee984883adca4b1fada1f685b5275ca0: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/110b4b780c48418fbd1dc18c489c5fce
I20240501 20:18:08.240401 17045 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:08.240957 14839 catalog_manager.cc:4830] TS ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715): tablet 110b4b780c48418fbd1dc18c489c5fce (table rebalance_test_table_1 [id=d544626e38bf4c95a54dda5d354fac7e]) successfully deleted
I20240501 20:18:08.241075 16051 raft_consensus.cc:1261] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Refusing update from remote peer 0021804537544af482a273d050ee9765: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:08.242266 17102 ts_tablet_manager.cc:1888] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:18:08.242489 17102 log.cc:1196] T 925816a53f1e45a4a81efd7dfddd2b9d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/925816a53f1e45a4a81efd7dfddd2b9d
I20240501 20:18:08.243089 17023 consensus_queue.cc:1035] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:08.243371 14840 catalog_manager.cc:4830] TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097): tablet 925816a53f1e45a4a81efd7dfddd2b9d (table rebalance_test_table_2 [id=7c1aa722484148d6b79edfcb01f3f400]) successfully deleted
I20240501 20:18:08.243849 17098 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } } }
I20240501 20:18:08.244403 14995 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P ee984883adca4b1fada1f685b5275ca0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } } }
I20240501 20:18:08.246227 16051 raft_consensus.cc:2932] T c32bb5190ebc49b5ad35a025746fa85d P 207d3f61d06541f5ae21e329b9fb5f8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } } }
I20240501 20:18:08.248661 14842 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c32bb5190ebc49b5ad35a025746fa85d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:08.249409 14855 catalog_manager.cc:5484] T c32bb5190ebc49b5ad35a025746fa85d P 0021804537544af482a273d050ee9765 reported cstate change: config changed from index 8 to 9, VOTER e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "0021804537544af482a273d050ee9765" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0021804537544af482a273d050ee9765" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207d3f61d06541f5ae21e329b9fb5f8e" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee984883adca4b1fada1f685b5275ca0" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 46715 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:08.253629 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet c32bb5190ebc49b5ad35a025746fa85d with delete_type TABLET_DATA_TOMBSTONED (TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40254
I20240501 20:18:08.254168 17102 tablet_replica.cc:330] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: stopping tablet replica
I20240501 20:18:08.254434 17102 raft_consensus.cc:2229] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:08.254849 17102 raft_consensus.cc:2258] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:08.255968 17102 ts_tablet_manager.cc:1875] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:08.261057 17102 ts_tablet_manager.cc:1888] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:18:08.261163 17102 log.cc:1196] T c32bb5190ebc49b5ad35a025746fa85d P e41b3aa39c7b4b0c90f3c36d6f07a0b4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/c32bb5190ebc49b5ad35a025746fa85d
I20240501 20:18:08.261840 14840 catalog_manager.cc:4830] TS e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097): tablet c32bb5190ebc49b5ad35a025746fa85d (table rebalance_test_table_0 [id=50169b2c2213457bb454bfe7b0ab73b1]) successfully deleted
I20240501 20:18:08.367785 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:08.377759 15901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:08.388388 15107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:08.397432 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:08.399281 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:08.544550 15238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:08.548727 16032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:08.551231 15901 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:08.560490 15107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:08.589296 14974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:08.787416 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 14914
I20240501 20:18:08.801851 15035 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:08.958209 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 14914
I20240501 20:18:08.984611 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15046
I20240501 20:18:08.990450 15167 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:18:09.156594 15061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:46715: connect: Connection refused (error 111)
W20240501 20:18:09.158434 15061 consensus_peers.cc:485] T b82e8ea1f3484534ae16e968fab04860 P e41b3aa39c7b4b0c90f3c36d6f07a0b4 -> Peer ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715): Couldn't send request to peer ee984883adca4b1fada1f685b5275ca0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:46715: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:09.163048 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15046
W20240501 20:18:09.188650 15195 connection.cc:535] server connection from 127.14.106.194:39839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:18:09.188632 15855 connection.cc:535] client connection to 127.14.106.194:41097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:18:09.188930 15855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:18:09.189411 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15176
W20240501 20:18:09.191220 15855 consensus_peers.cc:485] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e -> Peer e41b3aa39c7b4b0c90f3c36d6f07a0b4 (127.14.106.194:41097): Couldn't send request to peer e41b3aa39c7b4b0c90f3c36d6f07a0b4. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.194:41097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:09.196076 15299 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:18:09.262226 15854 consensus_peers.cc:485] T 3822c87b8cd44c4dbf432adf9aa0f590 P 79268a903afc4ef3a3bd79a32174fe2e -> Peer ee984883adca4b1fada1f685b5275ca0 (127.14.106.193:46715): Couldn't send request to peer ee984883adca4b1fada1f685b5275ca0. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:46715: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:18:09.346953 15193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.194:41097: connect: Connection refused (error 111)
I20240501 20:18:09.348225 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15176
W20240501 20:18:09.375464 15855 connection.cc:535] server connection from 127.14.106.195:57563 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:18:09.375919 14839 connection.cc:535] server connection from 127.14.106.195:43431 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:18:09.376247 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15970
I20240501 20:18:09.382387 16092 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:09.502948 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15970
I20240501 20:18:09.525875 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 15839
I20240501 20:18:09.532227 15961 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:09.653029 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 15839
I20240501 20:18:09.678534 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 14824
I20240501 20:18:09.679740 14885 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:09.825922 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 14824
2024-05-01T20:18:09Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (21291 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240501 20:18:09.885056 14763 test_util.cc:268] Using random seed: 795648870
I20240501 20:18:09.887090 14763 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:18:09.887153 14763 ts_itest-base.cc:113] --------------
I20240501 20:18:09.887217 14763 ts_itest-base.cc:114] 4 tablet servers
I20240501 20:18:09.887259 14763 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:18:09.887331 14763 ts_itest-base.cc:116] --------------
2024-05-01T20:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:18:09Z Disabled control of system clock
I20240501 20:18:09.941640 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:43831
--webserver_interface=127.14.106.254
--webserver_port=0
--builtin_ntp_servers=127.14.106.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.106.254:43831
--location_mapping_cmd=/tmp/dist-test-task8oH_XT/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 20:18:10.287076 17211 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:18:10.287416 17211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:10.287493 17211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:18:10.287564 17211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:10.295305 17211 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:18:10.295444 17211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:18:10.295526 17211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:18:10.295572 17211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:18:10.295639 17211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:18:10.304950 17211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task8oH_XT/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.14.106.254:43831
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.106.254:43831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:18:10.306247 17211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:18:10.307960 17211 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:18:10.318903 17219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:10.319357 17217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:10.319835 17216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:18:10.320647 17211 server_base.cc:1034] running on GCE node
I20240501 20:18:10.321385 17211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:18:10.322788 17211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:18:10.324012 17211 hybrid_clock.cc:648] HybridClock initialized: now 1714594690323943 us; error 40 us; skew 500 ppm
I20240501 20:18:10.324311 17211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:18:10.331966 17211 webserver.cc:460] Webserver started at http://127.14.106.254:42959/ using document root <none> and password file <none>
I20240501 20:18:10.332703 17211 fs_manager.cc:359] Metadata directory not provided
I20240501 20:18:10.332836 17211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:18:10.333144 17211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:18:10.335781 17211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "40e982ac4fa94fd2807ff1e1bc83831b"
format_stamp: "Formatted at 2024-05-01 20:18:10 on dist-test-slave-pg4x"
I20240501 20:18:10.336545 17211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "40e982ac4fa94fd2807ff1e1bc83831b"
format_stamp: "Formatted at 2024-05-01 20:18:10 on dist-test-slave-pg4x"
I20240501 20:18:10.341679 17211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:18:10.345162 17225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:10.346534 17211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:18:10.346740 17211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
uuid: "40e982ac4fa94fd2807ff1e1bc83831b"
format_stamp: "Formatted at 2024-05-01 20:18:10 on dist-test-slave-pg4x"
I20240501 20:18:10.346936 17211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:18:10.360284 17211 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:18:10.361150 17211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:18:10.361449 17211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:18:10.380542 17211 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.254:43831
I20240501 20:18:10.380578 17276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.254:43831 every 8 connection(s)
I20240501 20:18:10.381925 17211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:18:10.383493 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 17211
I20240501 20:18:10.383713 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:18:10.386801 17277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:10.397686 17277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b: Bootstrap starting.
I20240501 20:18:10.400748 17277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:10.401752 17277 log.cc:824] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:18:10.404525 17277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b: No bootstrap required, opened a new log
I20240501 20:18:10.410924 17277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e982ac4fa94fd2807ff1e1bc83831b" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 43831 } }
I20240501 20:18:10.411278 17277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:10.411378 17277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e982ac4fa94fd2807ff1e1bc83831b, State: Initialized, Role: FOLLOWER
I20240501 20:18:10.411994 17277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e982ac4fa94fd2807ff1e1bc83831b" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 43831 } }
I20240501 20:18:10.412209 17277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:18:10.412336 17277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:18:10.412488 17277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:10.414768 17277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e982ac4fa94fd2807ff1e1bc83831b" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 43831 } }
I20240501 20:18:10.415256 17277 leader_election.cc:302] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40e982ac4fa94fd2807ff1e1bc83831b; no voters:
I20240501 20:18:10.415848 17277 leader_election.cc:288] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:18:10.415987 17282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:10.416615 17282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [term 1 LEADER]: Becoming Leader. State: Replica: 40e982ac4fa94fd2807ff1e1bc83831b, State: Running, Role: LEADER
I20240501 20:18:10.417208 17282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e982ac4fa94fd2807ff1e1bc83831b" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 43831 } }
I20240501 20:18:10.417796 17277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:18:10.419795 17284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e982ac4fa94fd2807ff1e1bc83831b. Latest consensus state: current_term: 1 leader_uuid: "40e982ac4fa94fd2807ff1e1bc83831b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e982ac4fa94fd2807ff1e1bc83831b" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 43831 } } }
I20240501 20:18:10.420140 17284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [sys.catalog]: This master's current role is: LEADER
I20240501 20:18:10.420305 17283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40e982ac4fa94fd2807ff1e1bc83831b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e982ac4fa94fd2807ff1e1bc83831b" member_type: VOTER last_known_addr { host: "127.14.106.254" port: 43831 } } }
I20240501 20:18:10.426864 17283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b [sys.catalog]: This master's current role is: LEADER
I20240501 20:18:10.428138 17287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:18:10.434242 17287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:18:10.444386 17287 catalog_manager.cc:1347] Generated new cluster ID: 092a4197858c46b49991aadf9815835f
I20240501 20:18:10.444514 17287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:18:10.465592 17287 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:18:10.466794 17287 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:18:10.478332 17287 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 40e982ac4fa94fd2807ff1e1bc83831b: Generated new TSK 0
I20240501 20:18:10.479184 17287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:18:10.485617 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.193:0
--local_ip_for_outbound_sockets=127.14.106.193
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:43831
--builtin_ntp_servers=127.14.106.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:18:10.827965 17302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:10.828234 17302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:18:10.828384 17302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:10.835929 17302 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:18:10.836048 17302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:18:10.836225 17302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.193
I20240501 20:18:10.845367 17302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.106.193
--webserver_port=0
--tserver_master_addrs=127.14.106.254:43831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.193
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:18:10.846668 17302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:18:10.848380 17302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:18:10.860056 17307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:10.860534 17308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:10.860945 17310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:18:10.861810 17302 server_base.cc:1034] running on GCE node
I20240501 20:18:10.862381 17302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:18:10.863363 17302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:18:10.864578 17302 hybrid_clock.cc:648] HybridClock initialized: now 1714594690864533 us; error 47 us; skew 500 ppm
I20240501 20:18:10.864929 17302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:18:10.867225 17302 webserver.cc:460] Webserver started at http://127.14.106.193:41781/ using document root <none> and password file <none>
I20240501 20:18:10.867902 17302 fs_manager.cc:359] Metadata directory not provided
I20240501 20:18:10.867992 17302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:18:10.868252 17302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:18:10.870801 17302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "adc832fb6466498895e6f318e1c0ccb4"
format_stamp: "Formatted at 2024-05-01 20:18:10 on dist-test-slave-pg4x"
I20240501 20:18:10.871518 17302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "adc832fb6466498895e6f318e1c0ccb4"
format_stamp: "Formatted at 2024-05-01 20:18:10 on dist-test-slave-pg4x"
I20240501 20:18:10.876564 17302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 20:18:10.879925 17316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:10.881157 17302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:18:10.881312 17302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "adc832fb6466498895e6f318e1c0ccb4"
format_stamp: "Formatted at 2024-05-01 20:18:10 on dist-test-slave-pg4x"
I20240501 20:18:10.881518 17302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:18:10.911669 17302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:18:10.912541 17302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:18:10.912811 17302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:18:10.913758 17302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:18:10.915355 17302 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:18:10.915463 17302 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:10.915581 17302 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:18:10.915653 17302 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:10.957726 17302 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.193:35049
I20240501 20:18:10.957793 17428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.193:35049 every 8 connection(s)
I20240501 20:18:10.959074 17302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:18:10.963356 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 17302
I20240501 20:18:10.971982 17429 heartbeater.cc:347] Connected to a master server at 127.14.106.254:43831
I20240501 20:18:10.972295 17429 heartbeater.cc:464] Registering TS with master...
I20240501 20:18:10.973220 17429 heartbeater.cc:511] Master 127.14.106.254:43831 requested a full tablet report, sending...
I20240501 20:18:11.008392 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:18:11.012226 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.194:0
--local_ip_for_outbound_sockets=127.14.106.194
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:43831
--builtin_ntp_servers=127.14.106.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:18:11.022748 17242 ts_manager.cc:194] Registered new tserver with Master: adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:11.025394 17242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.193:35579
W20240501 20:18:11.345594 17434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:11.345870 17434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:18:11.345974 17434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:11.353502 17434 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:18:11.353639 17434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:18:11.353829 17434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.194
I20240501 20:18:11.362854 17434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.106.194
--webserver_port=0
--tserver_master_addrs=127.14.106.254:43831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.194
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:18:11.364184 17434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:18:11.365875 17434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:18:11.378131 17440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:11.378779 17442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:11.379400 17439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:18:11.380169 17434 server_base.cc:1034] running on GCE node
I20240501 20:18:11.380792 17434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:18:11.381767 17434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:18:11.383001 17434 hybrid_clock.cc:648] HybridClock initialized: now 1714594691382938 us; error 54 us; skew 500 ppm
I20240501 20:18:11.383337 17434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:18:11.385784 17434 webserver.cc:460] Webserver started at http://127.14.106.194:40491/ using document root <none> and password file <none>
I20240501 20:18:11.386659 17434 fs_manager.cc:359] Metadata directory not provided
I20240501 20:18:11.386788 17434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:18:11.387102 17434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:18:11.391695 17434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
format_stamp: "Formatted at 2024-05-01 20:18:11 on dist-test-slave-pg4x"
I20240501 20:18:11.392465 17434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
format_stamp: "Formatted at 2024-05-01 20:18:11 on dist-test-slave-pg4x"
I20240501 20:18:11.397810 17434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240501 20:18:11.401340 17448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:11.402688 17434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:18:11.402848 17434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
format_stamp: "Formatted at 2024-05-01 20:18:11 on dist-test-slave-pg4x"
I20240501 20:18:11.403046 17434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:18:11.421090 17434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:18:11.421871 17434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:18:11.422137 17434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:18:11.423110 17434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:18:11.424532 17434 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:18:11.424630 17434 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:11.424741 17434 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:18:11.424813 17434 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:11.456475 17434 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.194:37441
I20240501 20:18:11.456514 17560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.194:37441 every 8 connection(s)
I20240501 20:18:11.457733 17434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:18:11.465873 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 17434
I20240501 20:18:11.466014 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:18:11.469610 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.195:0
--local_ip_for_outbound_sockets=127.14.106.195
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:43831
--builtin_ntp_servers=127.14.106.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:18:11.470693 17561 heartbeater.cc:347] Connected to a master server at 127.14.106.254:43831
I20240501 20:18:11.471042 17561 heartbeater.cc:464] Registering TS with master...
I20240501 20:18:11.471856 17561 heartbeater.cc:511] Master 127.14.106.254:43831 requested a full tablet report, sending...
I20240501 20:18:11.519201 17242 ts_manager.cc:194] Registered new tserver with Master: d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:11.520514 17242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.194:46875
W20240501 20:18:11.813758 17565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:11.814055 17565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:18:11.814146 17565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:11.821731 17565 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:18:11.821851 17565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:18:11.821969 17565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.195
I20240501 20:18:11.830853 17565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.106.195
--webserver_port=0
--tserver_master_addrs=127.14.106.254:43831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.195
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:18:11.832166 17565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:18:11.833910 17565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:18:11.846005 17570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:11.846459 17571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:11.847141 17573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:18:11.847862 17565 server_base.cc:1034] running on GCE node
I20240501 20:18:11.848472 17565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:18:11.849404 17565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:18:11.850672 17565 hybrid_clock.cc:648] HybridClock initialized: now 1714594691850574 us; error 39 us; skew 500 ppm
I20240501 20:18:11.850989 17565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:18:11.853250 17565 webserver.cc:460] Webserver started at http://127.14.106.195:44743/ using document root <none> and password file <none>
I20240501 20:18:11.853969 17565 fs_manager.cc:359] Metadata directory not provided
I20240501 20:18:11.854077 17565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:18:11.854379 17565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:18:11.857003 17565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "be80849c49de4f188551b820fa459fd7"
format_stamp: "Formatted at 2024-05-01 20:18:11 on dist-test-slave-pg4x"
I20240501 20:18:11.857839 17565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "be80849c49de4f188551b820fa459fd7"
format_stamp: "Formatted at 2024-05-01 20:18:11 on dist-test-slave-pg4x"
I20240501 20:18:11.862859 17565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:18:11.866173 17579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:11.867496 17565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:18:11.867642 17565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "be80849c49de4f188551b820fa459fd7"
format_stamp: "Formatted at 2024-05-01 20:18:11 on dist-test-slave-pg4x"
I20240501 20:18:11.867836 17565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:18:11.889266 17565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:18:11.890172 17565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:18:11.890457 17565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:18:11.891474 17565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:18:11.892963 17565 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:18:11.893054 17565 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:11.893172 17565 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:18:11.893239 17565 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:11.926014 17565 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.195:43767
I20240501 20:18:11.926070 17691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.195:43767 every 8 connection(s)
I20240501 20:18:11.927366 17565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:18:11.936933 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 17565
I20240501 20:18:11.937132 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:18:11.940079 17692 heartbeater.cc:347] Connected to a master server at 127.14.106.254:43831
I20240501 20:18:11.940418 17692 heartbeater.cc:464] Registering TS with master...
I20240501 20:18:11.940769 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:0
--local_ip_for_outbound_sockets=127.14.106.196
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:43831
--builtin_ntp_servers=127.14.106.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:18:11.941458 17692 heartbeater.cc:511] Master 127.14.106.254:43831 requested a full tablet report, sending...
I20240501 20:18:11.988356 17242 ts_manager.cc:194] Registered new tserver with Master: be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:11.989291 17242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.195:47183
I20240501 20:18:12.029515 17429 heartbeater.cc:503] Master 127.14.106.254:43831 was elected leader, sending a full tablet report...
W20240501 20:18:12.281540 17696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:12.281827 17696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:18:12.281924 17696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:12.289522 17696 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:18:12.289640 17696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:18:12.289789 17696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:18:12.298676 17696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=0
--tserver_master_addrs=127.14.106.254:43831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:18:12.299963 17696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:18:12.301630 17696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:18:12.313959 17701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:12.314528 17704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:12.315145 17702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:18:12.319541 17696 server_base.cc:1034] running on GCE node
I20240501 20:18:12.320158 17696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:18:12.321079 17696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:18:12.322283 17696 hybrid_clock.cc:648] HybridClock initialized: now 1714594692322221 us; error 62 us; skew 500 ppm
I20240501 20:18:12.322605 17696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:18:12.324944 17696 webserver.cc:460] Webserver started at http://127.14.106.196:41835/ using document root <none> and password file <none>
I20240501 20:18:12.325688 17696 fs_manager.cc:359] Metadata directory not provided
I20240501 20:18:12.325804 17696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:18:12.326123 17696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:18:12.328774 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e03cb8ab898b48938fe49f167a721d54"
format_stamp: "Formatted at 2024-05-01 20:18:12 on dist-test-slave-pg4x"
I20240501 20:18:12.329514 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e03cb8ab898b48938fe49f167a721d54"
format_stamp: "Formatted at 2024-05-01 20:18:12 on dist-test-slave-pg4x"
I20240501 20:18:12.335008 17696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20240501 20:18:12.338438 17710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:12.339751 17696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:18:12.339905 17696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e03cb8ab898b48938fe49f167a721d54"
format_stamp: "Formatted at 2024-05-01 20:18:12 on dist-test-slave-pg4x"
I20240501 20:18:12.340116 17696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:18:12.359292 17696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:18:12.360107 17696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:18:12.360379 17696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:18:12.361335 17696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:18:12.362811 17696 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:18:12.362912 17696 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:12.363039 17696 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:18:12.363114 17696 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:12.396872 17696 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:34303
I20240501 20:18:12.396965 17822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:34303 every 8 connection(s)
I20240501 20:18:12.398208 17696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:18:12.408058 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 17696
I20240501 20:18:12.408223 14763 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:18:12.411214 17823 heartbeater.cc:347] Connected to a master server at 127.14.106.254:43831
I20240501 20:18:12.411566 17823 heartbeater.cc:464] Registering TS with master...
I20240501 20:18:12.412478 17823 heartbeater.cc:511] Master 127.14.106.254:43831 requested a full tablet report, sending...
I20240501 20:18:12.457260 17242 ts_manager.cc:194] Registered new tserver with Master: e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303)
I20240501 20:18:12.458226 17242 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:56143
I20240501 20:18:12.461725 14763 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:18:12.477469 14763 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:18:12.477623 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 17696
I20240501 20:18:12.485671 17818 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:12.523331 17561 heartbeater.cc:503] Master 127.14.106.254:43831 was elected leader, sending a full tablet report...
I20240501 20:18:12.567992 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 17696
I20240501 20:18:12.992136 17692 heartbeater.cc:503] Master 127.14.106.254:43831 was elected leader, sending a full tablet report...
I20240501 20:18:16.338173 17242 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38868:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:18:16.345445 17242 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:18:16.386811 17363 tablet_service.cc:1467] Processing CreateTablet for tablet f492ab8a54f24904ac46820a0cef5e2f (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.386823 17626 tablet_service.cc:1467] Processing CreateTablet for tablet f492ab8a54f24904ac46820a0cef5e2f (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.388526 17626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f492ab8a54f24904ac46820a0cef5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.388429 17362 tablet_service.cc:1467] Processing CreateTablet for tablet 53e5fb7d07044f62b5110548e3c6e32a (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.389353 17362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e5fb7d07044f62b5110548e3c6e32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.388870 17627 tablet_service.cc:1467] Processing CreateTablet for tablet 50117416c3494c9c85c504ccd7df747c (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.386879 17625 tablet_service.cc:1467] Processing CreateTablet for tablet 53e5fb7d07044f62b5110548e3c6e32a (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.391026 17627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50117416c3494c9c85c504ccd7df747c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.391283 17625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e5fb7d07044f62b5110548e3c6e32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.391198 17495 tablet_service.cc:1467] Processing CreateTablet for tablet f492ab8a54f24904ac46820a0cef5e2f (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.391582 17496 tablet_service.cc:1467] Processing CreateTablet for tablet 50117416c3494c9c85c504ccd7df747c (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.391978 17363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f492ab8a54f24904ac46820a0cef5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.392340 17495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f492ab8a54f24904ac46820a0cef5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.393047 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50117416c3494c9c85c504ccd7df747c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.394791 17364 tablet_service.cc:1467] Processing CreateTablet for tablet 50117416c3494c9c85c504ccd7df747c (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.390854 17494 tablet_service.cc:1467] Processing CreateTablet for tablet 53e5fb7d07044f62b5110548e3c6e32a (DEFAULT_TABLE table=rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:16.395686 17494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e5fb7d07044f62b5110548e3c6e32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.396493 17364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50117416c3494c9c85c504ccd7df747c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:16.412909 17846 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:16.417870 17846 tablet_bootstrap.cc:654] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.419251 17846 log.cc:824] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Log is configured to *not* fsync() on all Append() calls
I20240501 20:18:16.428421 17846 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:16.428862 17846 ts_tablet_manager.cc:1367] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240501 20:18:16.431353 17848 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:16.431808 17845 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:16.435166 17845 tablet_bootstrap.cc:654] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.435206 17848 tablet_bootstrap.cc:654] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.436242 17845 log.cc:824] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Log is configured to *not* fsync() on all Append() calls
I20240501 20:18:16.436431 17848 log.cc:824] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:18:16.438450 17846 raft_consensus.cc:348] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.438905 17846 raft_consensus.cc:374] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.439031 17846 raft_consensus.cc:729] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.439680 17848 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:16.439679 17845 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:16.439707 17846 consensus_queue.cc:260] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.439994 17845 ts_tablet_manager.cc:1367] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:18:16.439994 17848 ts_tablet_manager.cc:1367] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:18:16.441391 17846 ts_tablet_manager.cc:1398] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240501 20:18:16.441908 17846 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:16.443632 17846 tablet_bootstrap.cc:654] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.449888 17848 raft_consensus.cc:348] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.450532 17848 raft_consensus.cc:374] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.450701 17848 raft_consensus.cc:729] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.451182 17845 raft_consensus.cc:348] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.451545 17848 consensus_queue.cc:260] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.451736 17845 raft_consensus.cc:374] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.451892 17845 raft_consensus.cc:729] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.452392 17846 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:16.452571 17846 ts_tablet_manager.cc:1367] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 20:18:16.452870 17845 consensus_queue.cc:260] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.453434 17846 raft_consensus.cc:348] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.453714 17846 raft_consensus.cc:374] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.453848 17846 raft_consensus.cc:729] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.454118 17846 consensus_queue.cc:260] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.454692 17846 ts_tablet_manager.cc:1398] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:18:16.455134 17845 ts_tablet_manager.cc:1398] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20240501 20:18:16.455726 17845 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:16.458287 17845 tablet_bootstrap.cc:654] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.460110 17848 ts_tablet_manager.cc:1398] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.020s user 0.006s sys 0.013s
W20240501 20:18:16.462957 17562 tablet.cc:2369] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:18:16.464249 17848 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
W20240501 20:18:16.464569 17430 tablet.cc:2369] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:18:16.466805 17848 tablet_bootstrap.cc:654] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.469674 17854 raft_consensus.cc:482] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:16.470526 17846 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:16.471176 17854 raft_consensus.cc:504] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.473541 17846 tablet_bootstrap.cc:654] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.476732 17848 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:16.477023 17848 ts_tablet_manager.cc:1367] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240501 20:18:16.478400 17848 raft_consensus.cc:348] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.478801 17848 raft_consensus.cc:374] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.478977 17848 raft_consensus.cc:729] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.480037 17848 consensus_queue.cc:260] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.480983 17848 ts_tablet_manager.cc:1398] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:18:16.483736 17851 raft_consensus.cc:482] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:16.484002 17854 leader_election.cc:288] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:16.484088 17851 raft_consensus.cc:504] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.487481 17846 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:16.487727 17846 ts_tablet_manager.cc:1367] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.005s
I20240501 20:18:16.488862 17846 raft_consensus.cc:348] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.489316 17846 raft_consensus.cc:374] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.489485 17846 raft_consensus.cc:729] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.489845 17846 consensus_queue.cc:260] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.491782 17848 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:16.494141 17848 tablet_bootstrap.cc:654] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.497970 17845 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:16.498209 17845 ts_tablet_manager.cc:1367] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.032s
I20240501 20:18:16.499559 17845 raft_consensus.cc:348] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.499866 17845 raft_consensus.cc:374] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.500022 17845 raft_consensus.cc:729] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.500357 17845 consensus_queue.cc:260] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.501130 17845 ts_tablet_manager.cc:1398] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:18:16.505609 17845 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:16.507375 17845 tablet_bootstrap.cc:654] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:16.490207 17851 leader_election.cc:288] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:16.517617 17384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4" is_pre_election: true
I20240501 20:18:16.518296 17384 raft_consensus.cc:2454] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 0.
I20240501 20:18:16.519199 17845 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:16.519291 17449 leader_election.cc:302] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:16.519418 17845 ts_tablet_manager.cc:1367] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240501 20:18:16.519832 17855 raft_consensus.cc:2781] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:16.519994 17855 raft_consensus.cc:482] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:16.520124 17855 raft_consensus.cc:3037] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.520221 17845 raft_consensus.cc:348] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.520504 17845 raft_consensus.cc:374] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.520599 17845 raft_consensus.cc:729] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.523239 17845 consensus_queue.cc:260] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.523862 17845 ts_tablet_manager.cc:1398] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 20:18:16.531070 17855 raft_consensus.cc:504] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.532451 17855 leader_election.cc:288] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Requested vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:16.532613 17384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:16.532902 17384 raft_consensus.cc:3037] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.536769 17384 raft_consensus.cc:2454] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 1.
I20240501 20:18:16.537834 17449 leader_election.cc:302] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:16.538089 17846 ts_tablet_manager.cc:1398] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.050s user 0.000s sys 0.002s
I20240501 20:18:16.539578 17855 raft_consensus.cc:2781] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:16.539932 17855 raft_consensus.cc:686] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Becoming Leader. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Running, Role: LEADER
I20240501 20:18:16.540587 17855 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.555063 17647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7" is_pre_election: true
I20240501 20:18:16.555431 17242 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:16.556078 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7"
I20240501 20:18:16.556444 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50117416c3494c9c85c504ccd7df747c" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" is_pre_election: true
I20240501 20:18:16.556861 17516 raft_consensus.cc:2454] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 0.
W20240501 20:18:16.557005 17452 leader_election.cc:341] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:18:16.557559 17452 leader_election.cc:341] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:18:16.557936 17848 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:16.557854 17318 leader_election.cc:302] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:16.558022 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50117416c3494c9c85c504ccd7df747c" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7" is_pre_election: true
I20240501 20:18:16.558168 17848 ts_tablet_manager.cc:1367] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.066s user 0.018s sys 0.041s
I20240501 20:18:16.558324 17866 raft_consensus.cc:2781] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:16.558347 17646 raft_consensus.cc:2454] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 0.
I20240501 20:18:16.558455 17866 raft_consensus.cc:482] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:16.558773 17866 raft_consensus.cc:3037] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.558987 17848 raft_consensus.cc:348] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.559250 17848 raft_consensus.cc:374] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:16.559341 17848 raft_consensus.cc:729] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:16.559620 17848 consensus_queue.cc:260] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.560339 17848 ts_tablet_manager.cc:1398] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:18:16.561750 17866 raft_consensus.cc:504] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.562943 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50117416c3494c9c85c504ccd7df747c" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:16.562988 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50117416c3494c9c85c504ccd7df747c" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7"
I20240501 20:18:16.563162 17516 raft_consensus.cc:3037] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.563164 17646 raft_consensus.cc:3037] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.565384 17646 raft_consensus.cc:2454] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 1.
I20240501 20:18:16.565428 17516 raft_consensus.cc:2454] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 1.
I20240501 20:18:16.565781 17866 leader_election.cc:288] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 1 election: Requested vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:16.565819 17320 leader_election.cc:302] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7; no voters:
I20240501 20:18:16.566129 17866 raft_consensus.cc:2781] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:16.566323 17866 raft_consensus.cc:686] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Becoming Leader. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Running, Role: LEADER
I20240501 20:18:16.566797 17866 consensus_queue.cc:237] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.571244 17241 catalog_manager.cc:5484] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } }
W20240501 20:18:16.682783 17693 tablet.cc:2369] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:18:16.693277 17855 raft_consensus.cc:482] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:16.693483 17855 raft_consensus.cc:504] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.694257 17855 leader_election.cc:288] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:16.694833 17384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4" is_pre_election: true
I20240501 20:18:16.694916 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7" is_pre_election: true
I20240501 20:18:16.695080 17384 raft_consensus.cc:2454] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 0.
I20240501 20:18:16.695178 17646 raft_consensus.cc:2454] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 0.
I20240501 20:18:16.695544 17449 leader_election.cc:302] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:16.695941 17855 raft_consensus.cc:2781] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:16.696046 17855 raft_consensus.cc:482] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:16.696112 17855 raft_consensus.cc:3037] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.698012 17855 raft_consensus.cc:504] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.698568 17855 leader_election.cc:288] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Requested vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:16.699083 17384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:16.699144 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7"
I20240501 20:18:16.699337 17384 raft_consensus.cc:3037] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.699318 17646 raft_consensus.cc:3037] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.702136 17384 raft_consensus.cc:2454] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 1.
I20240501 20:18:16.702160 17646 raft_consensus.cc:2454] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 1.
I20240501 20:18:16.702606 17449 leader_election.cc:302] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:16.702934 17855 raft_consensus.cc:2781] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:16.703081 17855 raft_consensus.cc:686] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Becoming Leader. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Running, Role: LEADER
I20240501 20:18:16.703352 17855 consensus_queue.cc:237] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.706133 17241 catalog_manager.cc:5484] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:16.877811 17853 raft_consensus.cc:482] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:16.878049 17853 raft_consensus.cc:504] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:16.879707 17853 leader_election.cc:288] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:16.888337 17384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4" is_pre_election: true
I20240501 20:18:16.888388 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" is_pre_election: true
I20240501 20:18:16.888695 17384 raft_consensus.cc:2379] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be80849c49de4f188551b820fa459fd7 in current term 1: Already voted for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in this term.
I20240501 20:18:16.889427 17580 leader_election.cc:302] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be80849c49de4f188551b820fa459fd7; no voters: adc832fb6466498895e6f318e1c0ccb4, d66e5484e3dd4fcbaa55ef26dd16ee35
I20240501 20:18:16.889909 17853 raft_consensus.cc:3037] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:16.891930 17853 raft_consensus.cc:2726] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:18:16.949673 17855 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:16.957311 17855 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:18:17.103401 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:17.103765 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:17.111119 17870 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:18:17.127434 17866 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:17.141950 17890 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 20:18:17.157904 17855 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:17.162964 17854 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:18:17.330775 17449 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.106.193:35049, user_credentials={real_user=slave}} blocked reactor thread for 53106us
INSERT report
rows total: 2000
time total: 228.664 ms
time per row: 0.114332 ms
I20240501 20:18:17.497162 17240 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38868:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:18:17.497905 17240 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:18:17.506022 17494 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd75008254c4aef9740a5beb4f6288d (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.506247 17363 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd75008254c4aef9740a5beb4f6288d (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.506493 17494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd75008254c4aef9740a5beb4f6288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.506742 17363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd75008254c4aef9740a5beb4f6288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.507483 17627 tablet_service.cc:1467] Processing CreateTablet for tablet 49cf9d148ca64bdaa55fefb553dcb803 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.507918 17627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cf9d148ca64bdaa55fefb553dcb803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.508453 17625 tablet_service.cc:1467] Processing CreateTablet for tablet 16eb029403ad486f8f6da6b0d524baf0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.508817 17625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16eb029403ad486f8f6da6b0d524baf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.515235 17626 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd75008254c4aef9740a5beb4f6288d (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.515735 17626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd75008254c4aef9740a5beb4f6288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.518556 17935 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:17.519220 17496 tablet_service.cc:1467] Processing CreateTablet for tablet 16eb029403ad486f8f6da6b0d524baf0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.519286 17936 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:17.519610 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16eb029403ad486f8f6da6b0d524baf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.520761 17935 tablet_bootstrap.cc:654] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.521654 17936 tablet_bootstrap.cc:654] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.523483 17937 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:17.524300 17363 tablet_service.cc:1467] Processing CreateTablet for tablet 49cf9d148ca64bdaa55fefb553dcb803 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.524820 17363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cf9d148ca64bdaa55fefb553dcb803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.525627 17937 tablet_bootstrap.cc:654] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.527146 17362 tablet_service.cc:1467] Processing CreateTablet for tablet 16eb029403ad486f8f6da6b0d524baf0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.527681 17362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16eb029403ad486f8f6da6b0d524baf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.528002 17936 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:17.528156 17936 ts_tablet_manager.cc:1367] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240501 20:18:17.528939 17935 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:17.529081 17935 ts_tablet_manager.cc:1367] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 20:18:17.529053 17936 raft_consensus.cc:348] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.529311 17936 raft_consensus.cc:374] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.529438 17936 raft_consensus.cc:729] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.529757 17936 consensus_queue.cc:260] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.529858 17935 raft_consensus.cc:348] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.530077 17935 raft_consensus.cc:374] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.530176 17935 raft_consensus.cc:729] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.530364 17936 ts_tablet_manager.cc:1398] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:18:17.530496 17935 consensus_queue.cc:260] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.531141 17935 ts_tablet_manager.cc:1398] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:18:17.537133 17936 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:17.537505 17494 tablet_service.cc:1467] Processing CreateTablet for tablet 49cf9d148ca64bdaa55fefb553dcb803 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:17.538012 17494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cf9d148ca64bdaa55fefb553dcb803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:17.538878 17936 tablet_bootstrap.cc:654] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.540241 17935 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:17.541517 17937 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:17.541736 17937 ts_tablet_manager.cc:1367] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 20:18:17.542305 17935 tablet_bootstrap.cc:654] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.542694 17937 raft_consensus.cc:348] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.543002 17937 raft_consensus.cc:374] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.543143 17937 raft_consensus.cc:729] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.543460 17937 consensus_queue.cc:260] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.545222 17937 ts_tablet_manager.cc:1398] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:18:17.545475 17937 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:17.546821 17937 tablet_bootstrap.cc:654] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.547061 17936 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:17.547292 17936 ts_tablet_manager.cc:1367] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240501 20:18:17.548306 17936 raft_consensus.cc:348] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.549690 17936 raft_consensus.cc:374] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.549851 17936 raft_consensus.cc:729] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.550190 17936 consensus_queue.cc:260] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.551622 17936 ts_tablet_manager.cc:1398] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:18:17.552093 17936 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:17.554349 17936 tablet_bootstrap.cc:654] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.555029 17937 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:17.555275 17937 ts_tablet_manager.cc:1367] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240501 20:18:17.556124 17937 raft_consensus.cc:348] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.556406 17937 raft_consensus.cc:374] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.556542 17937 raft_consensus.cc:729] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.556797 17937 consensus_queue.cc:260] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.557097 17935 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:17.557379 17935 ts_tablet_manager.cc:1367] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20240501 20:18:17.557420 17937 ts_tablet_manager.cc:1398] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:18:17.557766 17937 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:17.558444 17935 raft_consensus.cc:348] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.558863 17935 raft_consensus.cc:374] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.559005 17935 raft_consensus.cc:729] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.559394 17937 tablet_bootstrap.cc:654] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.559417 17935 consensus_queue.cc:260] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.560536 17936 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:17.560756 17936 ts_tablet_manager.cc:1367] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240501 20:18:17.560972 17935 ts_tablet_manager.cc:1398] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:18:17.561311 17935 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:17.561616 17936 raft_consensus.cc:348] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.561862 17936 raft_consensus.cc:374] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.562008 17936 raft_consensus.cc:729] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.562325 17936 consensus_queue.cc:260] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.563014 17935 tablet_bootstrap.cc:654] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:17.563078 17936 ts_tablet_manager.cc:1398] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:18:17.564831 17937 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:17.564973 17937 ts_tablet_manager.cc:1367] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 20:18:17.565750 17937 raft_consensus.cc:348] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.565958 17937 raft_consensus.cc:374] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.566066 17937 raft_consensus.cc:729] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.566318 17937 consensus_queue.cc:260] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.566748 17935 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:17.566845 17937 ts_tablet_manager.cc:1398] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:18:17.566895 17935 ts_tablet_manager.cc:1367] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:18:17.567637 17935 raft_consensus.cc:348] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.567796 17935 raft_consensus.cc:374] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:17.567889 17935 raft_consensus.cc:729] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:17.568099 17935 consensus_queue.cc:260] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.568629 17935 ts_tablet_manager.cc:1398] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:18:17.614606 17923 raft_consensus.cc:482] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:17.614856 17923 raft_consensus.cc:504] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.615649 17923 leader_election.cc:288] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:17.616195 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd75008254c4aef9740a5beb4f6288d" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4" is_pre_election: true
I20240501 20:18:17.616199 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd75008254c4aef9740a5beb4f6288d" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7" is_pre_election: true
I20240501 20:18:17.616420 17383 raft_consensus.cc:2454] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 0.
I20240501 20:18:17.616439 17646 raft_consensus.cc:2454] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 0.
I20240501 20:18:17.616845 17449 leader_election.cc:302] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:17.617172 17923 raft_consensus.cc:2781] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:17.617295 17923 raft_consensus.cc:482] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:17.617395 17923 raft_consensus.cc:3037] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.619494 17923 raft_consensus.cc:504] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.620095 17923 leader_election.cc:288] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Requested vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:17.620570 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd75008254c4aef9740a5beb4f6288d" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:17.620625 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd75008254c4aef9740a5beb4f6288d" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7"
I20240501 20:18:17.620800 17383 raft_consensus.cc:3037] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.620834 17646 raft_consensus.cc:3037] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.623733 17646 raft_consensus.cc:2454] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 1.
I20240501 20:18:17.623749 17383 raft_consensus.cc:2454] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 1.
I20240501 20:18:17.624193 17452 leader_election.cc:302] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be80849c49de4f188551b820fa459fd7, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:17.624522 17923 raft_consensus.cc:2781] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:17.624670 17923 raft_consensus.cc:686] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Becoming Leader. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Running, Role: LEADER
I20240501 20:18:17.624974 17923 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.627847 17240 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:17.713141 17853 raft_consensus.cc:482] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:17.713382 17853 raft_consensus.cc:504] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.714094 17853 leader_election.cc:288] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:17.714720 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cf9d148ca64bdaa55fefb553dcb803" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" is_pre_election: true
I20240501 20:18:17.714776 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cf9d148ca64bdaa55fefb553dcb803" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4" is_pre_election: true
I20240501 20:18:17.714944 17516 raft_consensus.cc:2454] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 0.
I20240501 20:18:17.715016 17383 raft_consensus.cc:2454] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 0.
I20240501 20:18:17.715413 17581 leader_election.cc:302] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be80849c49de4f188551b820fa459fd7, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:17.715715 17853 raft_consensus.cc:2781] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:17.715818 17853 raft_consensus.cc:482] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:17.715909 17853 raft_consensus.cc:3037] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.717888 17853 raft_consensus.cc:504] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.718529 17853 leader_election.cc:288] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 election: Requested vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:17.719048 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cf9d148ca64bdaa55fefb553dcb803" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:17.719269 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cf9d148ca64bdaa55fefb553dcb803" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:17.719327 17383 raft_consensus.cc:3037] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.719439 17516 raft_consensus.cc:3037] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.721184 17383 raft_consensus.cc:2454] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 1.
I20240501 20:18:17.721282 17516 raft_consensus.cc:2454] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 1.
I20240501 20:18:17.721566 17580 leader_election.cc:302] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7; no voters:
I20240501 20:18:17.721856 17853 raft_consensus.cc:2781] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:17.722373 17853 raft_consensus.cc:686] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Becoming Leader. State: Replica: be80849c49de4f188551b820fa459fd7, State: Running, Role: LEADER
I20240501 20:18:17.722954 17853 consensus_queue.cc:237] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.726159 17240 catalog_manager.cc:5484] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to be80849c49de4f188551b820fa459fd7 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } }
I20240501 20:18:17.751485 17853 raft_consensus.cc:482] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:17.751667 17853 raft_consensus.cc:504] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.752323 17853 leader_election.cc:288] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:17.752846 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16eb029403ad486f8f6da6b0d524baf0" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" is_pre_election: true
I20240501 20:18:17.752900 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16eb029403ad486f8f6da6b0d524baf0" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4" is_pre_election: true
I20240501 20:18:17.753098 17516 raft_consensus.cc:2454] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 0.
I20240501 20:18:17.753114 17383 raft_consensus.cc:2454] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 0.
I20240501 20:18:17.753510 17581 leader_election.cc:302] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be80849c49de4f188551b820fa459fd7, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:17.753769 17853 raft_consensus.cc:2781] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:17.753887 17853 raft_consensus.cc:482] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:17.753979 17853 raft_consensus.cc:3037] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.755846 17853 raft_consensus.cc:504] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.756392 17853 leader_election.cc:288] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 election: Requested vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:17.756856 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16eb029403ad486f8f6da6b0d524baf0" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:17.756883 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16eb029403ad486f8f6da6b0d524baf0" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:17.757036 17383 raft_consensus.cc:3037] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.757053 17516 raft_consensus.cc:3037] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:17.758833 17516 raft_consensus.cc:2454] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 1.
I20240501 20:18:17.758911 17383 raft_consensus.cc:2454] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 1.
I20240501 20:18:17.759207 17581 leader_election.cc:302] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be80849c49de4f188551b820fa459fd7, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:17.759459 17853 raft_consensus.cc:2781] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:17.759581 17853 raft_consensus.cc:686] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Becoming Leader. State: Replica: be80849c49de4f188551b820fa459fd7, State: Running, Role: LEADER
I20240501 20:18:17.759884 17853 consensus_queue.cc:237] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:17.762717 17240 catalog_manager.cc:5484] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to be80849c49de4f188551b820fa459fd7 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } }
I20240501 20:18:18.011132 17923 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:18.018581 17855 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:18:18.125329 17951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:18.125641 17951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:18.133035 17951 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:18:18.204380 17853 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:18.209791 17948 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:18.217110 17948 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:18.231019 17948 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 250.163 ms
time per row: 0.125081 ms
I20240501 20:18:18.522974 17240 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38868:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:18:18.523665 17240 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:18:18.531219 17363 tablet_service.cc:1467] Processing CreateTablet for tablet d941cb6f2beb456d96fa262f0c86e868 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.531381 17494 tablet_service.cc:1467] Processing CreateTablet for tablet d941cb6f2beb456d96fa262f0c86e868 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.531718 17363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d941cb6f2beb456d96fa262f0c86e868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.531767 17494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d941cb6f2beb456d96fa262f0c86e868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.533231 17626 tablet_service.cc:1467] Processing CreateTablet for tablet d941cb6f2beb456d96fa262f0c86e868 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.533318 17627 tablet_service.cc:1467] Processing CreateTablet for tablet b31922f4bb7042f4bfc2a7bb03589e3c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.533720 17626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d941cb6f2beb456d96fa262f0c86e868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.534200 17627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31922f4bb7042f4bfc2a7bb03589e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.535117 17625 tablet_service.cc:1467] Processing CreateTablet for tablet 667551ec034c4f82ad9ddc89005fff41 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.535516 17625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667551ec034c4f82ad9ddc89005fff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.538472 17362 tablet_service.cc:1467] Processing CreateTablet for tablet 667551ec034c4f82ad9ddc89005fff41 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.538861 17362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667551ec034c4f82ad9ddc89005fff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.542492 17496 tablet_service.cc:1467] Processing CreateTablet for tablet 667551ec034c4f82ad9ddc89005fff41 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.542716 18015 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:18.542943 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667551ec034c4f82ad9ddc89005fff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.543375 18014 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:18.545105 18014 tablet_bootstrap.cc:654] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.545533 18015 tablet_bootstrap.cc:654] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.547147 17364 tablet_service.cc:1467] Processing CreateTablet for tablet b31922f4bb7042f4bfc2a7bb03589e3c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.547636 17364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31922f4bb7042f4bfc2a7bb03589e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.557008 17495 tablet_service.cc:1467] Processing CreateTablet for tablet b31922f4bb7042f4bfc2a7bb03589e3c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:18:18.557614 17495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31922f4bb7042f4bfc2a7bb03589e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:18.561470 18014 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:18.561723 18014 ts_tablet_manager.cc:1367] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.009s
I20240501 20:18:18.562677 18014 raft_consensus.cc:348] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.562937 18014 raft_consensus.cc:374] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.563097 18014 raft_consensus.cc:729] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.563442 18014 consensus_queue.cc:260] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.564102 18014 ts_tablet_manager.cc:1398] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:18:18.564567 18014 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:18.566196 18014 tablet_bootstrap.cc:654] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.571249 18017 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:18.571519 18015 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:18.571736 18015 ts_tablet_manager.cc:1367] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.011s
I20240501 20:18:18.572372 18014 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:18.572531 18014 ts_tablet_manager.cc:1367] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 20:18:18.572665 18015 raft_consensus.cc:348] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.572942 18015 raft_consensus.cc:374] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.573056 18015 raft_consensus.cc:729] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.573302 18014 raft_consensus.cc:348] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.573431 18017 tablet_bootstrap.cc:654] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.573472 18014 raft_consensus.cc:374] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.573312 18015 consensus_queue.cc:260] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.573552 18014 raft_consensus.cc:729] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.573824 18014 consensus_queue.cc:260] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.573959 18015 ts_tablet_manager.cc:1398] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:18:18.574394 18014 ts_tablet_manager.cc:1398] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:18:18.574838 18014 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:18.576503 18014 tablet_bootstrap.cc:654] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.577543 18017 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:18.577715 18017 ts_tablet_manager.cc:1367] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 20:18:18.578549 18017 raft_consensus.cc:348] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.578778 18017 raft_consensus.cc:374] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.578889 18017 raft_consensus.cc:729] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.579217 18015 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:18.580894 18015 tablet_bootstrap.cc:654] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.581148 18014 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4: No bootstrap required, opened a new log
I20240501 20:18:18.581269 18014 ts_tablet_manager.cc:1367] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:18:18.581822 18014 raft_consensus.cc:348] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.581790 18017 consensus_queue.cc:260] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.581967 18014 raft_consensus.cc:374] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.582048 18014 raft_consensus.cc:729] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.582253 18014 consensus_queue.cc:260] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.582397 18017 ts_tablet_manager.cc:1398] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:18:18.582723 18014 ts_tablet_manager.cc:1398] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 20:18:18.582854 18017 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:18.584398 18017 tablet_bootstrap.cc:654] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.587873 18015 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:18.588029 18015 ts_tablet_manager.cc:1367] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240501 20:18:18.588092 18017 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:18.588253 18017 ts_tablet_manager.cc:1367] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 20:18:18.588691 18015 raft_consensus.cc:348] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.588896 18015 raft_consensus.cc:374] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.588837 18017 raft_consensus.cc:348] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.588969 18015 raft_consensus.cc:729] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.589051 18017 raft_consensus.cc:374] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.589135 18017 raft_consensus.cc:729] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.589216 18015 consensus_queue.cc:260] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.589375 18017 consensus_queue.cc:260] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.589941 18017 ts_tablet_manager.cc:1398] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:18:18.590219 18017 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: Bootstrap starting.
I20240501 20:18:18.590415 18015 ts_tablet_manager.cc:1398] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:18:18.590727 18015 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:18.591921 18017 tablet_bootstrap.cc:654] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.591961 18015 tablet_bootstrap.cc:654] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7: Neither blocks nor log segments found. Creating new log.
I20240501 20:18:18.594798 17892 raft_consensus.cc:482] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:18.595017 17892 raft_consensus.cc:504] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.596040 17892 leader_election.cc:288] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:18.596807 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d941cb6f2beb456d96fa262f0c86e868" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7" is_pre_election: true
I20240501 20:18:18.596928 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d941cb6f2beb456d96fa262f0c86e868" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" is_pre_election: true
I20240501 20:18:18.597039 17646 raft_consensus.cc:2454] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 0.
I20240501 20:18:18.597113 17514 raft_consensus.cc:2454] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 0.
I20240501 20:18:18.597479 17320 leader_election.cc:302] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7; no voters:
I20240501 20:18:18.598726 17892 raft_consensus.cc:2781] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:18.598870 17892 raft_consensus.cc:482] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:18.598980 17892 raft_consensus.cc:3037] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.599901 18015 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7: No bootstrap required, opened a new log
I20240501 20:18:18.600026 18015 ts_tablet_manager.cc:1367] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20240501 20:18:18.600688 18015 raft_consensus.cc:348] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.600838 18015 raft_consensus.cc:374] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.600970 18015 raft_consensus.cc:729] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.601261 18015 consensus_queue.cc:260] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.601702 18017 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: No bootstrap required, opened a new log
I20240501 20:18:18.601800 18015 ts_tablet_manager.cc:1398] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:18:18.601897 18017 ts_tablet_manager.cc:1367] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 20:18:18.601925 17892 raft_consensus.cc:504] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.602744 17892 leader_election.cc:288] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 1 election: Requested vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:18.603183 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d941cb6f2beb456d96fa262f0c86e868" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7"
I20240501 20:18:18.603354 17646 raft_consensus.cc:3037] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.603703 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d941cb6f2beb456d96fa262f0c86e868" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:18.603901 17514 raft_consensus.cc:3037] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.603844 18017 raft_consensus.cc:348] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.604025 18017 raft_consensus.cc:374] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:18:18.604132 18017 raft_consensus.cc:729] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Initialized, Role: FOLLOWER
I20240501 20:18:18.604368 18017 consensus_queue.cc:260] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.604950 18017 ts_tablet_manager.cc:1398] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:18:18.606115 17646 raft_consensus.cc:2454] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 1.
I20240501 20:18:18.606608 17320 leader_election.cc:302] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7; no voters:
I20240501 20:18:18.606865 17514 raft_consensus.cc:2454] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 1.
I20240501 20:18:18.606959 17892 raft_consensus.cc:2781] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:18.607446 17892 raft_consensus.cc:686] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Becoming Leader. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Running, Role: LEADER
I20240501 20:18:18.607797 17892 consensus_queue.cc:237] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.610455 17240 catalog_manager.cc:5484] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193). New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:18.678755 17853 raft_consensus.cc:482] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:18.678927 17853 raft_consensus.cc:504] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.679668 17853 leader_election.cc:288] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:18.680135 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31922f4bb7042f4bfc2a7bb03589e3c" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4" is_pre_election: true
I20240501 20:18:18.680205 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31922f4bb7042f4bfc2a7bb03589e3c" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" is_pre_election: true
I20240501 20:18:18.680356 17383 raft_consensus.cc:2454] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 0.
I20240501 20:18:18.680490 17514 raft_consensus.cc:2454] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 0.
I20240501 20:18:18.680908 17580 leader_election.cc:302] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7; no voters:
I20240501 20:18:18.681177 17853 raft_consensus.cc:2781] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:18.681277 17853 raft_consensus.cc:482] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:18.681368 17853 raft_consensus.cc:3037] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.683452 17853 raft_consensus.cc:504] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.684031 17853 leader_election.cc:288] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 election: Requested vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:18.684397 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31922f4bb7042f4bfc2a7bb03589e3c" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:18.684464 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31922f4bb7042f4bfc2a7bb03589e3c" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:18.684608 17383 raft_consensus.cc:3037] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.684674 17514 raft_consensus.cc:3037] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.686759 17383 raft_consensus.cc:2454] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 1.
I20240501 20:18:18.686770 17514 raft_consensus.cc:2454] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 1.
I20240501 20:18:18.687144 17580 leader_election.cc:302] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7; no voters:
I20240501 20:18:18.687451 17853 raft_consensus.cc:2781] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:18.687595 17853 raft_consensus.cc:686] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Becoming Leader. State: Replica: be80849c49de4f188551b820fa459fd7, State: Running, Role: LEADER
I20240501 20:18:18.687872 17853 consensus_queue.cc:237] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.690511 17240 catalog_manager.cc:5484] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to be80849c49de4f188551b820fa459fd7 (127.14.106.195). New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } }
I20240501 20:18:18.899508 17923 raft_consensus.cc:482] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:18:18.899714 17923 raft_consensus.cc:504] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.900403 17923 leader_election.cc:288] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:18.900951 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4" is_pre_election: true
I20240501 20:18:18.900977 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7" is_pre_election: true
I20240501 20:18:18.901160 17383 raft_consensus.cc:2454] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 0.
I20240501 20:18:18.901203 17646 raft_consensus.cc:2454] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 0.
I20240501 20:18:18.901585 17449 leader_election.cc:302] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:18.901849 17923 raft_consensus.cc:2781] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:18:18.901954 17923 raft_consensus.cc:482] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:18:18.902020 17923 raft_consensus.cc:3037] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.903905 17923 raft_consensus.cc:504] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.904399 17923 leader_election.cc:288] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Requested vote from peers adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:18.904805 17383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:18.904929 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41" candidate_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be80849c49de4f188551b820fa459fd7"
I20240501 20:18:18.905025 17383 raft_consensus.cc:3037] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.905155 17646 raft_consensus.cc:3037] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:18:18.907922 17646 raft_consensus.cc:2454] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 1.
I20240501 20:18:18.907966 17383 raft_consensus.cc:2454] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66e5484e3dd4fcbaa55ef26dd16ee35 in term 1.
I20240501 20:18:18.908356 17452 leader_election.cc:302] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be80849c49de4f188551b820fa459fd7, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:18.908711 17923 raft_consensus.cc:2781] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:18:18.908859 17923 raft_consensus.cc:686] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Becoming Leader. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Running, Role: LEADER
I20240501 20:18:18.909097 17923 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:18.911787 17240 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194). New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:19.050209 17892 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:19.056011 17892 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:19.140684 17853 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:19.145236 17948 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:19.340394 17923 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:18:19.345162 17923 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:18:19.354748 18030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:19.355232 18030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:19.362857 18030 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 276.223 ms
time per row: 0.138111 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:18:20.714740 14763 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8oH_XT/build/asan/bin/kudu
/tmp/dist-test-task8oH_XT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.106.196:34303
--local_ip_for_outbound_sockets=127.14.106.196
--tserver_master_addrs=127.14.106.254:43831
--webserver_port=41835
--webserver_interface=127.14.106.196
--builtin_ntp_servers=127.14.106.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:18:21.057266 18090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:18:21.057549 18090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:18:21.057641 18090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:18:21.064958 18090 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:18:21.065097 18090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:18:21.065255 18090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.106.196
I20240501 20:18:21.074750 18090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.106.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.106.196:34303
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.106.196
--webserver_port=41835
--tserver_master_addrs=127.14.106.254:43831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.106.196
--log_dir=/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:18:21.076215 18090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:18:21.077986 18090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:18:21.090207 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:21.090680 18095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:18:21.091286 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:18:21.092082 18090 server_base.cc:1034] running on GCE node
I20240501 20:18:21.092846 18090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:18:21.093813 18090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:18:21.095052 18090 hybrid_clock.cc:648] HybridClock initialized: now 1714594701095003 us; error 51 us; skew 500 ppm
I20240501 20:18:21.095404 18090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:18:21.100987 18090 webserver.cc:460] Webserver started at http://127.14.106.196:41835/ using document root <none> and password file <none>
I20240501 20:18:21.101752 18090 fs_manager.cc:359] Metadata directory not provided
I20240501 20:18:21.101886 18090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:18:21.107276 18090 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:18:21.110352 18104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:21.111686 18090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:18:21.111860 18090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e03cb8ab898b48938fe49f167a721d54"
format_stamp: "Formatted at 2024-05-01 20:18:12 on dist-test-slave-pg4x"
I20240501 20:18:21.112668 18090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:18:21.132838 18090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:18:21.133610 18090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:18:21.133889 18090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:18:21.134850 18090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:18:21.136365 18090 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:18:21.136440 18090 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:21.136523 18090 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:18:21.136565 18090 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:18:21.169342 18090 rpc_server.cc:307] RPC server started. Bound to: 127.14.106.196:34303
I20240501 20:18:21.169381 18216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.106.196:34303 every 8 connection(s)
I20240501 20:18:21.170768 18090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:18:21.177014 14763 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8oH_XT/build/asan/bin/kudu as pid 18090
I20240501 20:18:21.184491 18217 heartbeater.cc:347] Connected to a master server at 127.14.106.254:43831
I20240501 20:18:21.184825 18217 heartbeater.cc:464] Registering TS with master...
I20240501 20:18:21.185905 18217 heartbeater.cc:511] Master 127.14.106.254:43831 requested a full tablet report, sending...
I20240501 20:18:21.188007 17240 ts_manager.cc:194] Re-registered known tserver with Master: e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303)
I20240501 20:18:21.189168 17240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.106.196:42169
I20240501 20:18:21.675540 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:21.679672 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:21.687541 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:21.691706 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:21.856309 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:21.866425 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:21.868621 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:21.877552 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:22.056615 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:22.063175 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:22.063647 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:22.071089 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:22.191908 18217 heartbeater.cc:503] Master 127.14.106.254:43831 was elected leader, sending a full tablet report...
I20240501 20:18:22.218825 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:22.222587 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:22.232112 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:22.232859 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:22.297008 17514 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.300330 17647 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:22.300951 17384 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:22.301060 18359 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.301527 17923 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.305079 18359 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.305826 17384 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.306988 17646 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
W20240501 20:18:22.308379 17449 consensus_peers.cc:485] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 667551ec034c4f82ad9ddc89005fff41. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:22.311205 17238 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:22.335752 17514 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.337827 17646 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:22.337888 17384 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:18:22.338210 17449 consensus_peers.cc:485] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fd75008254c4aef9740a5beb4f6288d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:22.338402 18359 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.338702 18358 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.341373 17923 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.342000 17384 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.342218 17646 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.346295 17240 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.367337 17514 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.369414 17384 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:18:22.369625 17449 consensus_peers.cc:485] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53e5fb7d07044f62b5110548e3c6e32a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:22.369691 17646 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:22.369925 18358 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.370247 18375 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.372807 18359 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.373467 17384 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.373616 17646 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.377907 17236 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:22.397403 17384 consensus_queue.cc:237] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.400182 17514 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:22.400985 18086 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.412101 18390 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.412480 17646 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:22.413421 18086 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.414412 17514 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.417672 17240 catalog_manager.cc:5484] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:18:22.425400 17317 consensus_peers.cc:485] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50117416c3494c9c85c504ccd7df747c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:22.426462 17646 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.442763 17646 consensus_queue.cc:237] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.445540 17514 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:22.446051 17384 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:22.446153 18400 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.446700 17948 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:22.449451 18400 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.450078 17514 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.451156 17384 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
W20240501 20:18:22.454535 17580 consensus_peers.cc:485] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49cf9d148ca64bdaa55fefb553dcb803. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:22.455650 17240 catalog_manager.cc:5484] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:22.683717 18430 ts_tablet_manager.cc:902] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:22.685695 18430 tablet_copy_client.cc:308] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:37441
I20240501 20:18:22.693071 17536 tablet_copy_service.cc:140] P d66e5484e3dd4fcbaa55ef26dd16ee35: Received BeginTabletCopySession request for tablet 667551ec034c4f82ad9ddc89005fff41 from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:56767)
I20240501 20:18:22.693311 17536 tablet_copy_service.cc:161] P d66e5484e3dd4fcbaa55ef26dd16ee35: Beginning new tablet copy session on tablet 667551ec034c4f82ad9ddc89005fff41 from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:56767: session id = e03cb8ab898b48938fe49f167a721d54-667551ec034c4f82ad9ddc89005fff41
I20240501 20:18:22.696023 17536 tablet_copy_source_session.cc:205] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:22.700565 18430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667551ec034c4f82ad9ddc89005fff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:22.711699 18430 tablet_copy_client.cc:783] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:22.712078 18430 tablet_copy_client.cc:647] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:22.714532 18430 tablet_copy_client.cc:523] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:22.718164 18430 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:22.730512 18430 log.cc:824] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: Log is configured to *not* fsync() on all Append() calls
I20240501 20:18:22.768028 18430 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:22.768548 18430 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:22.768973 18430 ts_tablet_manager.cc:1367] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.051s user 0.045s sys 0.004s
I20240501 20:18:22.773496 18430 raft_consensus.cc:348] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.773921 18430 raft_consensus.cc:729] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:22.774384 18430 consensus_queue.cc:260] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.775774 18430 ts_tablet_manager.cc:1398] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:18:22.776839 17536 tablet_copy_service.cc:342] P d66e5484e3dd4fcbaa55ef26dd16ee35: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-667551ec034c4f82ad9ddc89005fff41 received from {username='slave'} at 127.14.106.196:56767
I20240501 20:18:22.777065 17536 tablet_copy_service.cc:433] P d66e5484e3dd4fcbaa55ef26dd16ee35: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-667551ec034c4f82ad9ddc89005fff41 on tablet 667551ec034c4f82ad9ddc89005fff41 with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:22.854349 18430 ts_tablet_manager.cc:902] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:22.855793 18430 tablet_copy_client.cc:308] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:43767
I20240501 20:18:22.861568 17667 tablet_copy_service.cc:140] P be80849c49de4f188551b820fa459fd7: Received BeginTabletCopySession request for tablet 49cf9d148ca64bdaa55fefb553dcb803 from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:44057)
I20240501 20:18:22.861774 17667 tablet_copy_service.cc:161] P be80849c49de4f188551b820fa459fd7: Beginning new tablet copy session on tablet 49cf9d148ca64bdaa55fefb553dcb803 from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:44057: session id = e03cb8ab898b48938fe49f167a721d54-49cf9d148ca64bdaa55fefb553dcb803
I20240501 20:18:22.864499 17667 tablet_copy_source_session.cc:205] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:22.866166 18430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cf9d148ca64bdaa55fefb553dcb803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:22.870498 18430 tablet_copy_client.cc:783] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:22.870748 18430 tablet_copy_client.cc:647] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:22.872608 18430 tablet_copy_client.cc:523] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:22.873268 18468 ts_tablet_manager.cc:902] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:22.874941 18468 tablet_copy_client.cc:308] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:37441
I20240501 20:18:22.875607 17536 tablet_copy_service.cc:140] P d66e5484e3dd4fcbaa55ef26dd16ee35: Received BeginTabletCopySession request for tablet 3fd75008254c4aef9740a5beb4f6288d from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:56767)
I20240501 20:18:22.875820 17536 tablet_copy_service.cc:161] P d66e5484e3dd4fcbaa55ef26dd16ee35: Beginning new tablet copy session on tablet 3fd75008254c4aef9740a5beb4f6288d from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:56767: session id = e03cb8ab898b48938fe49f167a721d54-3fd75008254c4aef9740a5beb4f6288d
I20240501 20:18:22.876421 18430 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:22.878871 17536 tablet_copy_source_session.cc:205] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:22.880168 18468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd75008254c4aef9740a5beb4f6288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:22.885227 18473 ts_tablet_manager.cc:902] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:22.887084 18468 tablet_copy_client.cc:783] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:22.887379 18468 tablet_copy_client.cc:647] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:22.889717 18468 tablet_copy_client.cc:523] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:22.895012 18468 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:22.896317 18473 tablet_copy_client.cc:308] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:35049
I20240501 20:18:22.904372 17404 tablet_copy_service.cc:140] P adc832fb6466498895e6f318e1c0ccb4: Received BeginTabletCopySession request for tablet 50117416c3494c9c85c504ccd7df747c from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:52791)
I20240501 20:18:22.904628 17404 tablet_copy_service.cc:161] P adc832fb6466498895e6f318e1c0ccb4: Beginning new tablet copy session on tablet 50117416c3494c9c85c504ccd7df747c from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:52791: session id = e03cb8ab898b48938fe49f167a721d54-50117416c3494c9c85c504ccd7df747c
I20240501 20:18:22.907694 17404 tablet_copy_source_session.cc:205] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:22.909736 18473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50117416c3494c9c85c504ccd7df747c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:22.914988 18473 tablet_copy_client.cc:783] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:22.915359 18473 tablet_copy_client.cc:647] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:22.916550 18480 ts_tablet_manager.cc:902] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:22.918318 18473 tablet_copy_client.cc:523] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:22.920008 18430 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:22.920532 18430 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:22.920874 18430 ts_tablet_manager.cc:1367] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.045s user 0.044s sys 0.000s
I20240501 20:18:22.921983 18430 raft_consensus.cc:348] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.922398 18430 raft_consensus.cc:729] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:22.922741 18430 consensus_queue.cc:260] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.923509 18473 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:22.926278 18430 ts_tablet_manager.cc:1398] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:18:22.929344 17667 tablet_copy_service.cc:342] P be80849c49de4f188551b820fa459fd7: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-49cf9d148ca64bdaa55fefb553dcb803 received from {username='slave'} at 127.14.106.196:44057
I20240501 20:18:22.929518 17667 tablet_copy_service.cc:433] P be80849c49de4f188551b820fa459fd7: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-49cf9d148ca64bdaa55fefb553dcb803 on tablet 49cf9d148ca64bdaa55fefb553dcb803 with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:22.981607 18468 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:22.985316 18480 tablet_copy_client.cc:308] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:37441
I20240501 20:18:22.985641 18473 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:22.987166 18468 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:22.987502 17536 tablet_copy_service.cc:140] P d66e5484e3dd4fcbaa55ef26dd16ee35: Received BeginTabletCopySession request for tablet 53e5fb7d07044f62b5110548e3c6e32a from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:56767)
I20240501 20:18:22.987603 18468 ts_tablet_manager.cc:1367] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.093s user 0.043s sys 0.010s
I20240501 20:18:22.987659 18473 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:22.987697 17536 tablet_copy_service.cc:161] P d66e5484e3dd4fcbaa55ef26dd16ee35: Beginning new tablet copy session on tablet 53e5fb7d07044f62b5110548e3c6e32a from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:56767: session id = e03cb8ab898b48938fe49f167a721d54-53e5fb7d07044f62b5110548e3c6e32a
I20240501 20:18:22.987959 18473 ts_tablet_manager.cc:1367] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.065s user 0.041s sys 0.021s
I20240501 20:18:22.988448 18468 raft_consensus.cc:348] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.988713 18468 raft_consensus.cc:729] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:22.988924 18468 consensus_queue.cc:260] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.990223 18468 ts_tablet_manager.cc:1398] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:18:22.988924 18473 raft_consensus.cc:348] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.990810 18473 raft_consensus.cc:729] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:22.991021 18473 consensus_queue.cc:260] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:22.991540 18473 ts_tablet_manager.cc:1398] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:18:22.992183 17535 tablet_copy_service.cc:342] P d66e5484e3dd4fcbaa55ef26dd16ee35: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-3fd75008254c4aef9740a5beb4f6288d received from {username='slave'} at 127.14.106.196:56767
I20240501 20:18:22.992298 17535 tablet_copy_service.cc:433] P d66e5484e3dd4fcbaa55ef26dd16ee35: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-3fd75008254c4aef9740a5beb4f6288d on tablet 3fd75008254c4aef9740a5beb4f6288d with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:22.992275 17404 tablet_copy_service.cc:342] P adc832fb6466498895e6f318e1c0ccb4: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-50117416c3494c9c85c504ccd7df747c received from {username='slave'} at 127.14.106.196:52791
I20240501 20:18:22.992455 17404 tablet_copy_service.cc:433] P adc832fb6466498895e6f318e1c0ccb4: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-50117416c3494c9c85c504ccd7df747c on tablet 50117416c3494c9c85c504ccd7df747c with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:22.993574 17536 tablet_copy_source_session.cc:205] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:22.995654 18480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e5fb7d07044f62b5110548e3c6e32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:23.001542 18480 tablet_copy_client.cc:783] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:23.001928 18480 tablet_copy_client.cc:647] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:23.003834 18480 tablet_copy_client.cc:523] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:23.007369 18480 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:23.047374 18480 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:23.047788 18480 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:23.048056 18480 ts_tablet_manager.cc:1367] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.041s user 0.035s sys 0.005s
I20240501 20:18:23.048892 18480 raft_consensus.cc:348] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:23.049180 18480 raft_consensus.cc:729] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:23.049403 18480 consensus_queue.cc:260] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:23.050005 18480 ts_tablet_manager.cc:1398] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:18:23.050880 17536 tablet_copy_service.cc:342] P d66e5484e3dd4fcbaa55ef26dd16ee35: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-53e5fb7d07044f62b5110548e3c6e32a received from {username='slave'} at 127.14.106.196:56767
I20240501 20:18:23.051038 17536 tablet_copy_service.cc:433] P d66e5484e3dd4fcbaa55ef26dd16ee35: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-53e5fb7d07044f62b5110548e3c6e32a on tablet 53e5fb7d07044f62b5110548e3c6e32a with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:23.129374 18172 raft_consensus.cc:1203] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:23.254344 18172 raft_consensus.cc:1203] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:23.300241 18172 raft_consensus.cc:1203] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:23.332578 18172 raft_consensus.cc:1203] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:23.443246 18172 raft_consensus.cc:1203] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:23.713271 18513 raft_consensus.cc:1050] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:23.714409 18513 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:23.717180 17646 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.717340 18172 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.717363 17384 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.717953 18513 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.718355 18566 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.718706 18359 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.723065 18359 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.724161 18172 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.727017 17384 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.727970 17238 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:23.736387 18512 raft_consensus.cc:1050] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:23.737205 18512 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:23.737030 17646 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.739655 17647 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.739959 18172 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.740159 17384 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.741034 18513 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.741343 18566 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.743544 18566 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.743947 17515 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:23.745771 18171 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:23.746387 17646 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:23.746644 18566 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:23.746981 18359 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:23.749601 18513 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.750427 18566 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.751075 17647 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.755949 17645 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.757237 18172 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.757314 18171 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.761879 17383 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.762480 17240 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.767416 17238 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.768357 17227 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 667551ec034c4f82ad9ddc89005fff41 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:23.769088 17363 tablet_service.cc:1514] Processing DeleteTablet for tablet 667551ec034c4f82ad9ddc89005fff41 with delete_type TABLET_DATA_TOMBSTONED (TS adc832fb6466498895e6f318e1c0ccb4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32874
I20240501 20:18:23.781296 18582 tablet_replica.cc:330] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: stopping tablet replica
I20240501 20:18:23.782059 18582 raft_consensus.cc:2229] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:23.785408 17515 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:23.785746 18514 raft_consensus.cc:1050] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:23.786182 18582 raft_consensus.cc:2258] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:23.787539 18172 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:23.788154 18566 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:23.788678 18514 consensus_queue.cc:237] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:23.789359 17384 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:23.790020 18566 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:23.792316 18172 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.792397 18513 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.793219 18172 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.793483 17383 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.794395 18582 ts_tablet_manager.cc:1875] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:23.796540 17384 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.796916 17227 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fd75008254c4aef9740a5beb4f6288d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:23.797087 18581 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.797684 18581 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.797977 17236 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:23.804132 18582 ts_tablet_manager.cc:1888] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:23.804301 18582 log.cc:1196] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/667551ec034c4f82ad9ddc89005fff41
I20240501 20:18:23.804355 17626 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd75008254c4aef9740a5beb4f6288d with delete_type TABLET_DATA_TOMBSTONED (TS be80849c49de4f188551b820fa459fd7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42564
I20240501 20:18:23.805315 17226 catalog_manager.cc:4830] TS adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): tablet 667551ec034c4f82ad9ddc89005fff41 (table rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]) successfully deleted
I20240501 20:18:23.805727 17515 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.806308 18581 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.809238 18514 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.809957 17515 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.811321 17384 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.813263 18171 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.814244 17236 catalog_manager.cc:5484] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:23.816947 18591 tablet_replica.cc:330] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: stopping tablet replica
I20240501 20:18:23.817497 18591 raft_consensus.cc:2229] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:23.822258 17645 consensus_queue.cc:237] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:23.824091 17515 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:23.824156 18171 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:23.824609 18514 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:23.825026 18581 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:23.827857 18514 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.828994 18591 raft_consensus.cc:2258] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:23.832473 17229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49cf9d148ca64bdaa55fefb553dcb803 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:23.831786 17515 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.833519 18591 ts_tablet_manager.cc:1875] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:23.835352 17236 catalog_manager.cc:5484] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:23.840245 17363 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cf9d148ca64bdaa55fefb553dcb803 with delete_type TABLET_DATA_TOMBSTONED (TS adc832fb6466498895e6f318e1c0ccb4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32874
I20240501 20:18:23.840620 18582 tablet_replica.cc:330] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: stopping tablet replica
I20240501 20:18:23.840838 18582 raft_consensus.cc:2229] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:23.841094 18582 raft_consensus.cc:2258] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:23.842090 18582 ts_tablet_manager.cc:1875] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:23.844707 18591 ts_tablet_manager.cc:1888] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:23.844868 18591 log.cc:1196] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/3fd75008254c4aef9740a5beb4f6288d
I20240501 20:18:23.845923 17229 catalog_manager.cc:4830] TS be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767): tablet 3fd75008254c4aef9740a5beb4f6288d (table rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]) successfully deleted
I20240501 20:18:23.847525 18582 ts_tablet_manager.cc:1888] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:23.847656 18582 log.cc:1196] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/49cf9d148ca64bdaa55fefb553dcb803
I20240501 20:18:23.848400 17226 catalog_manager.cc:4830] TS adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): tablet 49cf9d148ca64bdaa55fefb553dcb803 (table rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]) successfully deleted
I20240501 20:18:23.855212 18463 raft_consensus.cc:1050] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:23.856189 18463 consensus_queue.cc:237] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:23.859243 18171 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.858872 18172 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.859896 17515 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.860644 17645 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:23.860705 18606 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.861207 18597 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.861845 18597 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:23.877570 18606 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.878868 18171 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.879456 17645 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.882261 17515 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.883194 17237 catalog_manager.cc:5484] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:23.889956 17384 consensus_queue.cc:237] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:23.892321 18171 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:23.892540 17515 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:23.892900 18463 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:23.893481 18607 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:23.895742 18607 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.896369 18171 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.897197 17514 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:23.898974 17226 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50117416c3494c9c85c504ccd7df747c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:23.901010 17236 catalog_manager.cc:5484] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:23.905344 17626 tablet_service.cc:1514] Processing DeleteTablet for tablet 50117416c3494c9c85c504ccd7df747c with delete_type TABLET_DATA_TOMBSTONED (TS be80849c49de4f188551b820fa459fd7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42564
I20240501 20:18:23.905966 18591 tablet_replica.cc:330] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: stopping tablet replica
I20240501 20:18:23.906167 18591 raft_consensus.cc:2229] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:23.906423 18591 raft_consensus.cc:2258] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:23.907428 18591 ts_tablet_manager.cc:1875] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:23.911870 18591 ts_tablet_manager.cc:1888] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:23.911988 18591 log.cc:1196] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/50117416c3494c9c85c504ccd7df747c
I20240501 20:18:23.912796 17229 catalog_manager.cc:4830] TS be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767): tablet 50117416c3494c9c85c504ccd7df747c (table rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]) successfully deleted
I20240501 20:18:23.930737 17384 consensus_queue.cc:237] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:23.933179 17645 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:23.933322 17514 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:23.933871 18607 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:18:23.934211 17317 consensus_peers.cc:485] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d941cb6f2beb456d96fa262f0c86e868. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:23.934221 18463 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:23.937239 18606 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:23.937857 17514 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:23.941033 17645 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:23.941900 17237 catalog_manager.cc:5484] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index -1 to 4, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:24.061743 18566 raft_consensus.cc:1050] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:24.062449 18566 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:24.064651 18171 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:24.064648 17384 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:24.064718 17645 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:24.065248 18566 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:24.065572 18580 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:24.065838 18512 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:24.069221 18580 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:24.069979 17384 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:24.070366 18171 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:24.074936 17236 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:24.076681 17645 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:24.079809 17514 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:24.081617 18171 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:24.082545 17645 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:24.082684 18566 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:24.083176 18566 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:24.085101 18359 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:24.086419 18171 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:24.087340 17645 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:24.088240 17227 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53e5fb7d07044f62b5110548e3c6e32a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:24.090181 17240 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 5 to 6, VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) evicted. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:24.094435 17363 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e5fb7d07044f62b5110548e3c6e32a with delete_type TABLET_DATA_TOMBSTONED (TS adc832fb6466498895e6f318e1c0ccb4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32874
I20240501 20:18:24.094810 18582 tablet_replica.cc:330] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: stopping tablet replica
I20240501 20:18:24.095010 18582 raft_consensus.cc:2229] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:24.095263 18582 raft_consensus.cc:2258] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:24.096091 18582 ts_tablet_manager.cc:1875] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:24.100354 18582 ts_tablet_manager.cc:1888] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:24.100466 18582 log.cc:1196] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-0/wal/wals/53e5fb7d07044f62b5110548e3c6e32a
I20240501 20:18:24.101044 17226 catalog_manager.cc:4830] TS adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): tablet 53e5fb7d07044f62b5110548e3c6e32a (table rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]) successfully deleted
I20240501 20:18:24.352244 18673 ts_tablet_manager.cc:902] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:24.353523 18673 tablet_copy_client.cc:308] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:35049
I20240501 20:18:24.354168 17404 tablet_copy_service.cc:140] P adc832fb6466498895e6f318e1c0ccb4: Received BeginTabletCopySession request for tablet d941cb6f2beb456d96fa262f0c86e868 from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:52791)
I20240501 20:18:24.354327 17404 tablet_copy_service.cc:161] P adc832fb6466498895e6f318e1c0ccb4: Beginning new tablet copy session on tablet d941cb6f2beb456d96fa262f0c86e868 from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:52791: session id = e03cb8ab898b48938fe49f167a721d54-d941cb6f2beb456d96fa262f0c86e868
I20240501 20:18:24.356426 17404 tablet_copy_source_session.cc:205] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:24.357729 18673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d941cb6f2beb456d96fa262f0c86e868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:24.361941 18673 tablet_copy_client.cc:783] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:24.362171 18673 tablet_copy_client.cc:647] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:24.363770 18673 tablet_copy_client.cc:523] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:24.366031 18673 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:24.396832 18673 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:24.397226 18673 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:24.397531 18673 ts_tablet_manager.cc:1367] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.032s user 0.031s sys 0.000s
I20240501 20:18:24.398260 18673 raft_consensus.cc:348] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:24.398453 18673 raft_consensus.cc:729] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:24.398703 18673 consensus_queue.cc:260] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:24.399197 18673 ts_tablet_manager.cc:1398] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 20:18:24.400023 17404 tablet_copy_service.cc:342] P adc832fb6466498895e6f318e1c0ccb4: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-d941cb6f2beb456d96fa262f0c86e868 received from {username='slave'} at 127.14.106.196:52791
I20240501 20:18:24.400149 17404 tablet_copy_service.cc:433] P adc832fb6466498895e6f318e1c0ccb4: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-d941cb6f2beb456d96fa262f0c86e868 on tablet d941cb6f2beb456d96fa262f0c86e868 with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:24.870612 18171 raft_consensus.cc:1203] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:18:25.457902 18695 raft_consensus.cc:1050] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:25.458775 18695 consensus_queue.cc:237] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:25.461267 18171 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:25.461717 17645 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:25.461968 18706 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:25.462596 18706 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:25.471989 18695 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:25.472690 17514 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:25.473032 17645 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:25.475672 18171 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:25.476347 18695 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:25.476840 17240 catalog_manager.cc:5484] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:25.480510 17514 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:25.986800 17384 consensus_queue.cc:237] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:25.988777 18171 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:25.988832 17514 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:25.990293 18733 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:25.991129 18734 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:25.993854 18734 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:25.994597 18171 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:25.995563 17514 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:25.997702 17226 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d941cb6f2beb456d96fa262f0c86e868 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:25.999492 17237 catalog_manager.cc:5484] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 5 to 6, VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) evicted. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:26.003674 17626 tablet_service.cc:1514] Processing DeleteTablet for tablet d941cb6f2beb456d96fa262f0c86e868 with delete_type TABLET_DATA_TOMBSTONED (TS be80849c49de4f188551b820fa459fd7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42564
I20240501 20:18:26.004331 18739 tablet_replica.cc:330] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: stopping tablet replica
I20240501 20:18:26.004546 18739 raft_consensus.cc:2229] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:26.004882 18739 raft_consensus.cc:2258] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:26.005709 18739 ts_tablet_manager.cc:1875] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:26.010432 18739 ts_tablet_manager.cc:1888] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:18:26.010526 18739 log.cc:1196] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-2/wal/wals/d941cb6f2beb456d96fa262f0c86e868
I20240501 20:18:26.011165 17229 catalog_manager.cc:4830] TS be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767): tablet d941cb6f2beb456d96fa262f0c86e868 (table rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]) successfully deleted
I20240501 20:18:26.217814 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:26.231109 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:26.239647 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:26.240787 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:26.398113 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:26.398806 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:26.400283 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:26.404089 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:26.559940 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:26.560613 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:26.566473 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:26.581676 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:27.160144 17236 ts_manager.cc:292] Set tserver state for d66e5484e3dd4fcbaa55ef26dd16ee35 to MAINTENANCE_MODE
I20240501 20:18:27.722153 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:27.741730 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:27.747262 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:27.766924 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:27.915452 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:27.918318 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:27.933750 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:27.935519 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:28.206002 17363 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:18:28.217540 17626 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:18:28.230751 18152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:28.236889 17494 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:28.315135 17514 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:28.317432 18172 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:28.317741 17645 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:28.318256 18962 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:28.318614 18813 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:28.321256 18962 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.321902 17645 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.322268 18172 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.326380 17237 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:28.340976 17514 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
I20240501 20:18:28.343093 17645 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:28.343194 18172 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:18:28.344182 17449 consensus_peers.cc:485] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 -> Peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): Couldn't send request to peer adc832fb6466498895e6f318e1c0ccb4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:28.344372 18962 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:28.344851 18813 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:28.345297 17514 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:28.347577 18172 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:28.347592 17645 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:28.347430 18965 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:28.348189 18172 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:28.350466 18978 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:28.350669 17227 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 53e5fb7d07044f62b5110548e3c6e32a with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:28.351243 18813 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:28.353739 17236 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:28.360015 17645 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.360150 18813 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.360795 18171 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.365360 17236 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.372162 17645 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:28.376308 17514 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
W20240501 20:18:28.379245 17449 consensus_peers.cc:485] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 -> Peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): Couldn't send request to peer adc832fb6466498895e6f318e1c0ccb4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:28.379362 18171 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:28.379585 17645 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:28.379968 18962 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:28.380447 18977 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:28.382743 18962 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:28.383352 18171 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:28.388236 17227 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 667551ec034c4f82ad9ddc89005fff41 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:28.388903 17237 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:28.392638 17645 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:28.402299 17514 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:28.404438 18171 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:28.404536 17384 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:28.404973 18978 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:28.405254 18977 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:28.408217 18978 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.409219 18171 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.414944 17236 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.418015 17384 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:28.425721 17514 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:28.428408 17384 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:18:28.428434 17452 consensus_peers.cc:485] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 -> Peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767): Couldn't send request to peer be80849c49de4f188551b820fa459fd7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:28.429060 18977 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:28.429098 18171 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:28.429749 18977 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:28.431970 18977 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:28.435220 17227 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3fd75008254c4aef9740a5beb4f6288d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:28.435861 17384 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:28.437350 18171 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:28.437645 17236 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:28.445811 17514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fd75008254c4aef9740a5beb4f6288d"
dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39094
I20240501 20:18:28.446223 17514 raft_consensus.cc:595] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:28.770668 19025 ts_tablet_manager.cc:902] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Initiating tablet copy from peer d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:28.772236 19025 tablet_copy_client.cc:308] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:37441
I20240501 20:18:28.773121 17536 tablet_copy_service.cc:140] P d66e5484e3dd4fcbaa55ef26dd16ee35: Received BeginTabletCopySession request for tablet 667551ec034c4f82ad9ddc89005fff41 from peer adc832fb6466498895e6f318e1c0ccb4 ({username='slave'} at 127.14.106.193:43285)
I20240501 20:18:28.773294 17536 tablet_copy_service.cc:161] P d66e5484e3dd4fcbaa55ef26dd16ee35: Beginning new tablet copy session on tablet 667551ec034c4f82ad9ddc89005fff41 from peer adc832fb6466498895e6f318e1c0ccb4 at {username='slave'} at 127.14.106.193:43285: session id = adc832fb6466498895e6f318e1c0ccb4-667551ec034c4f82ad9ddc89005fff41
I20240501 20:18:28.775285 17536 tablet_copy_source_session.cc:205] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:28.776608 19025 ts_tablet_manager.cc:1875] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:28.781086 19025 ts_tablet_manager.cc:1888] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:28.781328 19025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667551ec034c4f82ad9ddc89005fff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:28.783815 19025 tablet_copy_client.cc:783] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:28.784111 19025 tablet_copy_client.cc:647] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:28.786096 19025 tablet_copy_client.cc:523] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:28.788890 19025 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:28.821069 19025 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:28.821569 19025 tablet_bootstrap.cc:492] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap complete.
I20240501 20:18:28.821852 19025 ts_tablet_manager.cc:1367] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.008s
I20240501 20:18:28.822500 19025 raft_consensus.cc:348] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
I20240501 20:18:28.822798 19025 raft_consensus.cc:729] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: LEARNER
I20240501 20:18:28.822961 19025 consensus_queue.cc:260] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
I20240501 20:18:28.823467 19025 ts_tablet_manager.cc:1398] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:18:28.824326 17536 tablet_copy_service.cc:342] P d66e5484e3dd4fcbaa55ef26dd16ee35: Request end of tablet copy session adc832fb6466498895e6f318e1c0ccb4-667551ec034c4f82ad9ddc89005fff41 received from {username='slave'} at 127.14.106.193:43285
I20240501 20:18:28.824450 17536 tablet_copy_service.cc:433] P d66e5484e3dd4fcbaa55ef26dd16ee35: ending tablet copy session adc832fb6466498895e6f318e1c0ccb4-667551ec034c4f82ad9ddc89005fff41 on tablet 667551ec034c4f82ad9ddc89005fff41 with peer adc832fb6466498895e6f318e1c0ccb4
I20240501 20:18:28.897006 18965 raft_consensus.cc:979] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: : Instructing follower e03cb8ab898b48938fe49f167a721d54 to start an election
I20240501 20:18:28.897203 18965 raft_consensus.cc:1067] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Signalling peer e03cb8ab898b48938fe49f167a721d54 to start an election
I20240501 20:18:28.898187 18171 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fd75008254c4aef9740a5beb4f6288d"
dest_uuid: "e03cb8ab898b48938fe49f167a721d54"
from {username='slave'} at 127.14.106.194:45653
I20240501 20:18:28.898500 18171 raft_consensus.cc:482] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:18:28.898744 18171 raft_consensus.cc:3037] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:28.901317 18171 raft_consensus.cc:504] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:28.903208 18171 leader_election.cc:288] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [CANDIDATE]: Term 2 election: Requested vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:28.903820 17384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd75008254c4aef9740a5beb4f6288d" candidate_uuid: "e03cb8ab898b48938fe49f167a721d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:28.904049 17384 raft_consensus.cc:3037] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:28.904110 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd75008254c4aef9740a5beb4f6288d" candidate_uuid: "e03cb8ab898b48938fe49f167a721d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:28.904351 17514 raft_consensus.cc:3032] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:18:28.904431 17514 raft_consensus.cc:729] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Running, Role: LEADER
I20240501 20:18:28.904790 17514 consensus_queue.cc:260] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:28.905249 17514 raft_consensus.cc:3037] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:28.907413 17514 raft_consensus.cc:2454] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e03cb8ab898b48938fe49f167a721d54 in term 2.
I20240501 20:18:28.907550 17384 raft_consensus.cc:2454] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e03cb8ab898b48938fe49f167a721d54 in term 2.
I20240501 20:18:28.908284 18106 leader_election.cc:302] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d66e5484e3dd4fcbaa55ef26dd16ee35, e03cb8ab898b48938fe49f167a721d54; no voters:
I20240501 20:18:28.908478 19036 ts_tablet_manager.cc:902] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Initiating tablet copy from peer d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:28.908691 18966 raft_consensus.cc:2781] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:18:28.909130 18966 raft_consensus.cc:686] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 2 LEADER]: Becoming Leader. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Running, Role: LEADER
I20240501 20:18:28.909673 18966 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:28.913568 17236 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 reported cstate change: term changed from 1 to 2, leader changed from d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) to e03cb8ab898b48938fe49f167a721d54 (127.14.106.196). New cstate: current_term: 2 leader_uuid: "e03cb8ab898b48938fe49f167a721d54" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:28.915138 19036 tablet_copy_client.cc:308] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:37441
I20240501 20:18:28.916910 17536 tablet_copy_service.cc:140] P d66e5484e3dd4fcbaa55ef26dd16ee35: Received BeginTabletCopySession request for tablet 3fd75008254c4aef9740a5beb4f6288d from peer be80849c49de4f188551b820fa459fd7 ({username='slave'} at 127.14.106.195:41283)
I20240501 20:18:28.917071 17536 tablet_copy_service.cc:161] P d66e5484e3dd4fcbaa55ef26dd16ee35: Beginning new tablet copy session on tablet 3fd75008254c4aef9740a5beb4f6288d from peer be80849c49de4f188551b820fa459fd7 at {username='slave'} at 127.14.106.195:41283: session id = be80849c49de4f188551b820fa459fd7-3fd75008254c4aef9740a5beb4f6288d
I20240501 20:18:28.919692 19025 ts_tablet_manager.cc:902] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Initiating tablet copy from peer d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:28.919728 17536 tablet_copy_source_session.cc:205] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:28.921214 19036 ts_tablet_manager.cc:1875] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:28.926923 19036 ts_tablet_manager.cc:1888] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:28.927502 19036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd75008254c4aef9740a5beb4f6288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:28.931077 19036 tablet_copy_client.cc:783] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:28.931368 19036 tablet_copy_client.cc:647] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:28.933982 19036 tablet_copy_client.cc:523] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:28.939416 19036 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:28.939604 19025 tablet_copy_client.cc:308] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:37441
I20240501 20:18:28.940399 17536 tablet_copy_service.cc:140] P d66e5484e3dd4fcbaa55ef26dd16ee35: Received BeginTabletCopySession request for tablet 53e5fb7d07044f62b5110548e3c6e32a from peer adc832fb6466498895e6f318e1c0ccb4 ({username='slave'} at 127.14.106.193:43285)
I20240501 20:18:28.940629 17536 tablet_copy_service.cc:161] P d66e5484e3dd4fcbaa55ef26dd16ee35: Beginning new tablet copy session on tablet 53e5fb7d07044f62b5110548e3c6e32a from peer adc832fb6466498895e6f318e1c0ccb4 at {username='slave'} at 127.14.106.193:43285: session id = adc832fb6466498895e6f318e1c0ccb4-53e5fb7d07044f62b5110548e3c6e32a
I20240501 20:18:28.943754 17536 tablet_copy_source_session.cc:205] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:28.944738 19025 ts_tablet_manager.cc:1875] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:28.950883 19025 ts_tablet_manager.cc:1888] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:28.951119 19025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e5fb7d07044f62b5110548e3c6e32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:28.953152 19025 tablet_copy_client.cc:783] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:28.953346 19025 tablet_copy_client.cc:647] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:28.954849 19025 tablet_copy_client.cc:523] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:28.957007 19025 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:28.979440 19036 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:28.979843 19036 tablet_bootstrap.cc:492] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Bootstrap complete.
I20240501 20:18:28.980096 19036 ts_tablet_manager.cc:1367] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.041s user 0.027s sys 0.008s
I20240501 20:18:28.980758 19036 raft_consensus.cc:348] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:28.980978 19036 raft_consensus.cc:729] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: LEARNER
I20240501 20:18:28.981173 19036 consensus_queue.cc:260] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:28.981729 19036 ts_tablet_manager.cc:1398] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:18:28.982618 17536 tablet_copy_service.cc:342] P d66e5484e3dd4fcbaa55ef26dd16ee35: Request end of tablet copy session be80849c49de4f188551b820fa459fd7-3fd75008254c4aef9740a5beb4f6288d received from {username='slave'} at 127.14.106.195:41283
I20240501 20:18:28.982777 17536 tablet_copy_service.cc:433] P d66e5484e3dd4fcbaa55ef26dd16ee35: ending tablet copy session be80849c49de4f188551b820fa459fd7-3fd75008254c4aef9740a5beb4f6288d on tablet 3fd75008254c4aef9740a5beb4f6288d with peer be80849c49de4f188551b820fa459fd7
I20240501 20:18:28.998709 19025 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:28.999228 19025 tablet_bootstrap.cc:492] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Bootstrap complete.
I20240501 20:18:28.999459 19025 ts_tablet_manager.cc:1367] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.043s user 0.041s sys 0.002s
I20240501 20:18:29.000041 19025 raft_consensus.cc:348] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
I20240501 20:18:29.000221 19025 raft_consensus.cc:729] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: LEARNER
I20240501 20:18:29.000385 19025 consensus_queue.cc:260] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
I20240501 20:18:29.000794 19025 ts_tablet_manager.cc:1398] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:18:29.001538 17536 tablet_copy_service.cc:342] P d66e5484e3dd4fcbaa55ef26dd16ee35: Request end of tablet copy session adc832fb6466498895e6f318e1c0ccb4-53e5fb7d07044f62b5110548e3c6e32a received from {username='slave'} at 127.14.106.193:43285
I20240501 20:18:29.001652 17536 tablet_copy_service.cc:433] P d66e5484e3dd4fcbaa55ef26dd16ee35: ending tablet copy session adc832fb6466498895e6f318e1c0ccb4-53e5fb7d07044f62b5110548e3c6e32a on tablet 53e5fb7d07044f62b5110548e3c6e32a with peer adc832fb6466498895e6f318e1c0ccb4
I20240501 20:18:29.289729 17514 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Refusing update from remote peer e03cb8ab898b48938fe49f167a721d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:18:29.290403 18966 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:29.294953 17384 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Refusing update from remote peer e03cb8ab898b48938fe49f167a721d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:18:29.295001 17645 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 2 LEARNER]: Refusing update from remote peer e03cb8ab898b48938fe49f167a721d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:18:29.295820 18966 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:29.296150 19037 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:29.300494 19037 raft_consensus.cc:1050] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54: attempting to promote NON_VOTER be80849c49de4f188551b820fa459fd7 to VOTER
I20240501 20:18:29.302001 19037 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } }
I20240501 20:18:29.304214 17384 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Refusing update from remote peer e03cb8ab898b48938fe49f167a721d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:29.304526 17514 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Refusing update from remote peer e03cb8ab898b48938fe49f167a721d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:29.304724 19080 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:29.305051 17645 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 2 LEARNER]: Refusing update from remote peer e03cb8ab898b48938fe49f167a721d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:29.305126 18966 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:29.305773 18966 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:29.307317 19080 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:29.307999 17384 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:29.311543 17645 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:29.311547 17514 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:29.312651 17237 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 8 to 10, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e03cb8ab898b48938fe49f167a721d54" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:29.324682 18171 consensus_queue.cc:237] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } }
I20240501 20:18:29.326303 17384 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Refusing update from remote peer e03cb8ab898b48938fe49f167a721d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:18:29.326794 17645 raft_consensus.cc:1261] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Refusing update from remote peer e03cb8ab898b48938fe49f167a721d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:18:29.326799 19080 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:18:29.327252 19076 consensus_queue.cc:1035] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:18:29.329077 19080 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:29.329639 17384 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:29.329844 17645 raft_consensus.cc:2932] T 3fd75008254c4aef9740a5beb4f6288d P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:29.332284 17226 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fd75008254c4aef9740a5beb4f6288d with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:29.334004 17236 catalog_manager.cc:5484] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 10 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "e03cb8ab898b48938fe49f167a721d54" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:29.338405 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd75008254c4aef9740a5beb4f6288d with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:29.339288 19082 tablet_replica.cc:330] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:29.339771 19082 raft_consensus.cc:2229] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:29.340137 19082 raft_consensus.cc:2258] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:29.341495 19082 ts_tablet_manager.cc:1875] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:29.345932 19082 ts_tablet_manager.cc:1888] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 20:18:29.346102 19082 log.cc:1196] T 3fd75008254c4aef9740a5beb4f6288d P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/3fd75008254c4aef9740a5beb4f6288d
I20240501 20:18:29.346957 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet 3fd75008254c4aef9740a5beb4f6288d (table rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]) successfully deleted
I20240501 20:18:29.360204 17384 raft_consensus.cc:1203] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:18:29.544128 17384 raft_consensus.cc:1203] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:18:29.619406 17645 consensus_queue.cc:237] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:29.621502 17514 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:29.621867 17384 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:29.622094 18581 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:29.622514 19105 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:29.625270 18581 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:29.625800 17384 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:29.626230 17514 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:29.631178 17237 catalog_manager.cc:5484] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:29.636796 17645 consensus_queue.cc:237] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
W20240501 20:18:29.638947 17580 consensus_peers.cc:485] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b31922f4bb7042f4bfc2a7bb03589e3c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:29.639907 17514 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:29.640400 19105 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:29.641131 17384 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:29.641789 19105 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:29.642992 19105 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:29.646785 17229 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b31922f4bb7042f4bfc2a7bb03589e3c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:29.646477 17384 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:29.647877 17514 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:29.649310 17240 catalog_manager.cc:5484] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:29.810225 18965 raft_consensus.cc:1050] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: attempting to promote NON_VOTER adc832fb6466498895e6f318e1c0ccb4 to VOTER
I20240501 20:18:29.810968 18965 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:29.813154 18171 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:29.813202 17645 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:29.813223 17384 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:29.813824 19126 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:29.814173 18962 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:29.814442 18965 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:29.818279 18962 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:29.819522 18171 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:29.822409 17645 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:29.822664 17384 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:29.823343 17237 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:29.904181 17514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41"
dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39224
I20240501 20:18:29.904368 17514 raft_consensus.cc:595] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:30.023023 19150 ts_tablet_manager.cc:902] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:30.024113 19150 tablet_copy_client.cc:308] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:43767
I20240501 20:18:30.024747 17667 tablet_copy_service.cc:140] P be80849c49de4f188551b820fa459fd7: Received BeginTabletCopySession request for tablet b31922f4bb7042f4bfc2a7bb03589e3c from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:44057)
I20240501 20:18:30.024935 17667 tablet_copy_service.cc:161] P be80849c49de4f188551b820fa459fd7: Beginning new tablet copy session on tablet b31922f4bb7042f4bfc2a7bb03589e3c from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:44057: session id = e03cb8ab898b48938fe49f167a721d54-b31922f4bb7042f4bfc2a7bb03589e3c
I20240501 20:18:30.026975 17667 tablet_copy_source_session.cc:205] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:30.028127 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31922f4bb7042f4bfc2a7bb03589e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:30.032701 19150 tablet_copy_client.cc:783] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:30.032951 19150 tablet_copy_client.cc:647] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:30.034529 19150 tablet_copy_client.cc:523] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:30.037034 19150 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:30.066716 19150 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:30.067126 19150 tablet_bootstrap.cc:492] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:30.067380 19150 ts_tablet_manager.cc:1367] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.004s
I20240501 20:18:30.068011 19150 raft_consensus.cc:348] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:30.068229 19150 raft_consensus.cc:729] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:30.068418 19150 consensus_queue.cc:260] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:30.068931 19150 ts_tablet_manager.cc:1398] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:18:30.069885 17667 tablet_copy_service.cc:342] P be80849c49de4f188551b820fa459fd7: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-b31922f4bb7042f4bfc2a7bb03589e3c received from {username='slave'} at 127.14.106.196:44057
I20240501 20:18:30.070047 17667 tablet_copy_service.cc:433] P be80849c49de4f188551b820fa459fd7: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-b31922f4bb7042f4bfc2a7bb03589e3c on tablet b31922f4bb7042f4bfc2a7bb03589e3c with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:30.140535 17514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41"
dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39262
I20240501 20:18:30.140707 17514 raft_consensus.cc:595] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:30.162288 18965 raft_consensus.cc:1050] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: attempting to promote NON_VOTER adc832fb6466498895e6f318e1c0ccb4 to VOTER
I20240501 20:18:30.162930 18965 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:30.164870 17384 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:30.164932 18171 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:30.165104 17645 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:30.165493 18962 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:30.165817 19126 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:30.166111 18979 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:30.168952 19126 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:30.169674 17645 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:30.173168 18171 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:30.173563 17384 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:30.176512 17237 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:30.313057 18979 raft_consensus.cc:979] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: : Instructing follower be80849c49de4f188551b820fa459fd7 to start an election
I20240501 20:18:30.313243 19126 raft_consensus.cc:1067] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Signalling peer be80849c49de4f188551b820fa459fd7 to start an election
I20240501 20:18:30.314018 17645 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41"
dest_uuid: "be80849c49de4f188551b820fa459fd7"
from {username='slave'} at 127.14.106.194:43755
I20240501 20:18:30.314275 17645 raft_consensus.cc:482] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:18:30.314364 17645 raft_consensus.cc:3037] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:30.316394 17645 raft_consensus.cc:504] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:30.317246 17645 leader_election.cc:288] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 2 election: Requested vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303), adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:30.317798 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e03cb8ab898b48938fe49f167a721d54"
I20240501 20:18:30.317888 17384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
I20240501 20:18:30.317939 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667551ec034c4f82ad9ddc89005fff41" candidate_uuid: "be80849c49de4f188551b820fa459fd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:30.318106 18171 raft_consensus.cc:3037] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:30.318111 17384 raft_consensus.cc:3037] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:30.318158 17514 raft_consensus.cc:3032] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:18:30.318264 17514 raft_consensus.cc:729] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Running, Role: LEADER
I20240501 20:18:30.318742 17514 consensus_queue.cc:260] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:30.319368 17514 raft_consensus.cc:3037] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:30.321173 17384 raft_consensus.cc:2454] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 2.
I20240501 20:18:30.321308 17514 raft_consensus.cc:2454] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 2.
I20240501 20:18:30.321770 17581 leader_election.cc:302] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:30.322139 19108 raft_consensus.cc:2781] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:18:30.322108 18171 raft_consensus.cc:2454] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be80849c49de4f188551b820fa459fd7 in term 2.
I20240501 20:18:30.322746 19108 raft_consensus.cc:686] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 2 LEADER]: Becoming Leader. State: Replica: be80849c49de4f188551b820fa459fd7, State: Running, Role: LEADER
I20240501 20:18:30.323115 19108 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:30.325768 17236 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 reported cstate change: term changed from 1 to 2, leader changed from d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) to be80849c49de4f188551b820fa459fd7 (127.14.106.195). New cstate: current_term: 2 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:30.367847 17514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a"
dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39286
I20240501 20:18:30.368073 17514 raft_consensus.cc:595] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:30.544790 18171 raft_consensus.cc:1203] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:18:30.605240 17514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a"
dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39300
I20240501 20:18:30.605480 17514 raft_consensus.cc:595] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:30.678122 19126 raft_consensus.cc:979] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: : Instructing follower adc832fb6466498895e6f318e1c0ccb4 to start an election
I20240501 20:18:30.678296 19126 raft_consensus.cc:1067] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Signalling peer adc832fb6466498895e6f318e1c0ccb4 to start an election
I20240501 20:18:30.679000 17384 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a"
dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
from {username='slave'} at 127.14.106.194:36253
I20240501 20:18:30.679267 17384 raft_consensus.cc:482] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:18:30.679388 17384 raft_consensus.cc:3037] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:30.681414 17384 raft_consensus.cc:504] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:30.682291 17384 leader_election.cc:288] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 2 election: Requested vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767), e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303)
I20240501 20:18:30.682667 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:30.682780 17645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "be80849c49de4f188551b820fa459fd7"
I20240501 20:18:30.682875 17514 raft_consensus.cc:3032] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:18:30.682965 17514 raft_consensus.cc:729] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Running, Role: LEADER
I20240501 20:18:30.683013 17645 raft_consensus.cc:3037] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:30.682945 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e5fb7d07044f62b5110548e3c6e32a" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e03cb8ab898b48938fe49f167a721d54"
I20240501 20:18:30.683171 18171 raft_consensus.cc:3037] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:30.683223 17514 consensus_queue.cc:260] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:30.683665 17514 raft_consensus.cc:3037] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:30.685736 17514 raft_consensus.cc:2454] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 2.
I20240501 20:18:30.685850 17645 raft_consensus.cc:2454] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 2.
I20240501 20:18:30.686053 18171 raft_consensus.cc:2454] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 2.
I20240501 20:18:30.686434 17320 leader_election.cc:302] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7, d66e5484e3dd4fcbaa55ef26dd16ee35; no voters:
I20240501 20:18:30.686764 18733 raft_consensus.cc:2781] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:18:30.687142 18733 raft_consensus.cc:686] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 2 LEADER]: Becoming Leader. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Running, Role: LEADER
I20240501 20:18:30.687422 18733 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:30.690055 17240 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: term changed from 1 to 2, leader changed from d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) to adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193). New cstate: current_term: 2 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:30.726238 17384 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:30.726826 19201 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:30.743741 18171 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:30.744225 19201 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:30.748317 17514 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:30.749190 19201 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:30.751984 17645 consensus_queue.cc:237] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:30.757380 17384 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:18:30.757916 19201 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:18:30.758299 18171 raft_consensus.cc:1261] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:18:30.759166 19201 consensus_queue.cc:1035] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:18:30.760947 19180 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:30.761658 17384 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:30.763943 17229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 667551ec034c4f82ad9ddc89005fff41 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:30.764456 18171 raft_consensus.cc:2932] T 667551ec034c4f82ad9ddc89005fff41 P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:30.765084 17240 catalog_manager.cc:5484] T 667551ec034c4f82ad9ddc89005fff41 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 9 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:30.769047 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet 667551ec034c4f82ad9ddc89005fff41 with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:30.771381 19227 tablet_replica.cc:330] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:30.771592 19227 raft_consensus.cc:2229] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:30.771778 19227 raft_consensus.cc:2258] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:30.772714 19227 ts_tablet_manager.cc:1875] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:30.776940 19227 ts_tablet_manager.cc:1888] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 20:18:30.777094 19227 log.cc:1196] T 667551ec034c4f82ad9ddc89005fff41 P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/667551ec034c4f82ad9ddc89005fff41
I20240501 20:18:30.777812 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet 667551ec034c4f82ad9ddc89005fff41 (table rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]) successfully deleted
I20240501 20:18:31.057528 17645 consensus_queue.cc:237] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:31.059478 17384 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:31.059477 17514 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:31.060071 19201 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:31.060315 19180 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:31.063313 19108 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:31.063925 17514 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:31.064023 17384 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:31.065842 19201 raft_consensus.cc:1050] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:31.066694 19201 consensus_queue.cc:237] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:31.069337 17240 catalog_manager.cc:5484] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:31.070677 18171 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:31.071218 17515 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:31.071259 19108 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:31.071991 19180 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:31.074419 17384 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:31.075229 19108 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:31.077646 17645 consensus_queue.cc:237] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:31.079470 17514 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:18:31.079859 17580 consensus_peers.cc:485] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16eb029403ad486f8f6da6b0d524baf0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:31.080673 17384 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:31.080243 19201 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:31.080665 19180 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.080982 17383 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.084378 17514 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.084621 18172 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.087302 17515 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:31.088474 19180 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:18:31.090485 17236 catalog_manager.cc:5484] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.086812 19108 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:31.098132 17237 catalog_manager.cc:5484] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:31.098732 17229 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 16eb029403ad486f8f6da6b0d524baf0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:31.105487 17383 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:31.107532 17645 consensus_queue.cc:237] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:31.110214 18171 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:31.110734 17384 raft_consensus.cc:1261] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:31.111783 19257 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:31.112051 19201 consensus_queue.cc:1035] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:31.114501 19257 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.116117 17384 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.117125 18171 raft_consensus.cc:2932] T b31922f4bb7042f4bfc2a7bb03589e3c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.120759 18172 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:31.121289 19222 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:31.121546 17229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b31922f4bb7042f4bfc2a7bb03589e3c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:31.122660 17237 catalog_manager.cc:5484] T b31922f4bb7042f4bfc2a7bb03589e3c P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 6 to 7, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.127468 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet b31922f4bb7042f4bfc2a7bb03589e3c with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:31.127856 19227 tablet_replica.cc:330] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:31.128099 19227 raft_consensus.cc:2229] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:31.128347 19227 raft_consensus.cc:2258] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:31.129350 19227 ts_tablet_manager.cc:1875] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:31.134202 19227 ts_tablet_manager.cc:1888] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:18:31.134284 19227 log.cc:1196] T b31922f4bb7042f4bfc2a7bb03589e3c P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/b31922f4bb7042f4bfc2a7bb03589e3c
I20240501 20:18:31.135143 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet b31922f4bb7042f4bfc2a7bb03589e3c (table rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]) successfully deleted
I20240501 20:18:31.136245 17515 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:31.136814 19222 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:31.140811 17645 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:18:31.141392 18733 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:31.372573 17384 consensus_queue.cc:237] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:31.374693 18171 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:31.374701 17515 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:31.375299 19268 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:31.375622 19222 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:31.378096 19268 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.378765 17515 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.379065 18171 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.383318 17237 catalog_manager.cc:5484] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:31.389670 17384 consensus_queue.cc:237] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:31.391587 18171 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:18:31.391917 17320 consensus_peers.cc:485] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 -> Peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767): Couldn't send request to peer be80849c49de4f188551b820fa459fd7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:31.392078 19268 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:31.392841 17515 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:31.393421 19268 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:31.395395 19222 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:31.398694 17226 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d941cb6f2beb456d96fa262f0c86e868 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:31.398417 17515 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:31.400945 17240 catalog_manager.cc:5484] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:31.403416 18172 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:31.528863 19297 ts_tablet_manager.cc:902] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:31.530217 19297 tablet_copy_client.cc:308] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:43767
I20240501 20:18:31.530929 17667 tablet_copy_service.cc:140] P be80849c49de4f188551b820fa459fd7: Received BeginTabletCopySession request for tablet 16eb029403ad486f8f6da6b0d524baf0 from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:44057)
I20240501 20:18:31.531102 17667 tablet_copy_service.cc:161] P be80849c49de4f188551b820fa459fd7: Beginning new tablet copy session on tablet 16eb029403ad486f8f6da6b0d524baf0 from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:44057: session id = e03cb8ab898b48938fe49f167a721d54-16eb029403ad486f8f6da6b0d524baf0
I20240501 20:18:31.533207 17667 tablet_copy_source_session.cc:205] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:31.534379 19297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16eb029403ad486f8f6da6b0d524baf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:31.538863 19297 tablet_copy_client.cc:783] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:31.539093 19297 tablet_copy_client.cc:647] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:31.540692 19297 tablet_copy_client.cc:523] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:31.542933 19297 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:31.573261 19297 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:31.573648 19297 tablet_bootstrap.cc:492] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:31.573904 19297 ts_tablet_manager.cc:1367] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20240501 20:18:31.574563 19297 raft_consensus.cc:348] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:31.574810 19297 raft_consensus.cc:729] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:31.574997 19297 consensus_queue.cc:260] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:31.575450 19297 ts_tablet_manager.cc:1398] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 20:18:31.576257 17667 tablet_copy_service.cc:342] P be80849c49de4f188551b820fa459fd7: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-16eb029403ad486f8f6da6b0d524baf0 received from {username='slave'} at 127.14.106.196:44057
I20240501 20:18:31.576439 17667 tablet_copy_service.cc:433] P be80849c49de4f188551b820fa459fd7: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-16eb029403ad486f8f6da6b0d524baf0 on tablet 16eb029403ad486f8f6da6b0d524baf0 with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:31.717324 17383 consensus_queue.cc:237] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:31.719336 17646 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:18:31.719348 18171 raft_consensus.cc:1261] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:18:31.719924 19222 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:18:31.720216 19268 consensus_queue.cc:1035] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:18:31.722376 18733 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:31.723057 17646 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:31.725431 17226 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53e5fb7d07044f62b5110548e3c6e32a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:31.726483 18171 raft_consensus.cc:2932] T 53e5fb7d07044f62b5110548e3c6e32a P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:31.727020 17236 catalog_manager.cc:5484] T 53e5fb7d07044f62b5110548e3c6e32a P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 9 to 11, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:31.730763 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e5fb7d07044f62b5110548e3c6e32a with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:31.731436 19320 tablet_replica.cc:330] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:31.731652 19320 raft_consensus.cc:2229] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:31.731882 19320 raft_consensus.cc:2258] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:31.732767 19320 ts_tablet_manager.cc:1875] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:31.737550 19320 ts_tablet_manager.cc:1888] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 20:18:31.737668 19320 log.cc:1196] T 53e5fb7d07044f62b5110548e3c6e32a P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/53e5fb7d07044f62b5110548e3c6e32a
I20240501 20:18:31.738420 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet 53e5fb7d07044f62b5110548e3c6e32a (table rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]) successfully deleted
I20240501 20:18:31.886852 19331 ts_tablet_manager.cc:902] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Initiating tablet copy from peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:31.889385 19331 tablet_copy_client.cc:308] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:35049
I20240501 20:18:31.890089 17404 tablet_copy_service.cc:140] P adc832fb6466498895e6f318e1c0ccb4: Received BeginTabletCopySession request for tablet d941cb6f2beb456d96fa262f0c86e868 from peer be80849c49de4f188551b820fa459fd7 ({username='slave'} at 127.14.106.195:53919)
I20240501 20:18:31.890235 17404 tablet_copy_service.cc:161] P adc832fb6466498895e6f318e1c0ccb4: Beginning new tablet copy session on tablet d941cb6f2beb456d96fa262f0c86e868 from peer be80849c49de4f188551b820fa459fd7 at {username='slave'} at 127.14.106.195:53919: session id = be80849c49de4f188551b820fa459fd7-d941cb6f2beb456d96fa262f0c86e868
I20240501 20:18:31.893229 17404 tablet_copy_source_session.cc:205] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:31.894860 19331 ts_tablet_manager.cc:1875] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:31.899493 19331 ts_tablet_manager.cc:1888] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:31.899727 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d941cb6f2beb456d96fa262f0c86e868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:31.901727 19331 tablet_copy_client.cc:783] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:31.901943 19331 tablet_copy_client.cc:647] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:31.903550 19331 tablet_copy_client.cc:523] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:31.905769 19331 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:31.938759 19331 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:31.939182 19331 tablet_bootstrap.cc:492] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Bootstrap complete.
I20240501 20:18:31.939460 19331 ts_tablet_manager.cc:1367] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.034s user 0.035s sys 0.000s
I20240501 20:18:31.940117 19331 raft_consensus.cc:348] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:31.940341 19331 raft_consensus.cc:729] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: LEARNER
I20240501 20:18:31.940542 19331 consensus_queue.cc:260] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:31.941066 19331 ts_tablet_manager.cc:1398] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:18:31.941913 17404 tablet_copy_service.cc:342] P adc832fb6466498895e6f318e1c0ccb4: Request end of tablet copy session be80849c49de4f188551b820fa459fd7-d941cb6f2beb456d96fa262f0c86e868 received from {username='slave'} at 127.14.106.195:53919
I20240501 20:18:31.942045 17404 tablet_copy_service.cc:433] P adc832fb6466498895e6f318e1c0ccb4: ending tablet copy session be80849c49de4f188551b820fa459fd7-d941cb6f2beb456d96fa262f0c86e868 on tablet d941cb6f2beb456d96fa262f0c86e868 with peer be80849c49de4f188551b820fa459fd7
I20240501 20:18:32.061802 18171 raft_consensus.cc:1203] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:18:32.107195 17515 consensus_queue.cc:237] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } }
I20240501 20:18:32.110071 17383 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:32.110050 17646 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:18:32.110682 19126 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:32.111016 19349 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:18:32.113788 19126 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:32.114254 17383 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:32.114602 17646 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } }
I20240501 20:18:32.119391 17236 catalog_manager.cc:5484] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } }
I20240501 20:18:32.125358 17515 consensus_queue.cc:237] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:32.127735 17383 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:32.127874 17646 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:18:32.128619 19352 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:18:32.129140 17449 consensus_peers.cc:485] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 -> Peer e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303): Couldn't send request to peer e03cb8ab898b48938fe49f167a721d54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f492ab8a54f24904ac46820a0cef5e2f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:32.129360 19349 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:18:32.131260 19352 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:32.132187 17646 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:32.132174 17383 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } } }
I20240501 20:18:32.136400 17227 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f492ab8a54f24904ac46820a0cef5e2f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:32.137204 17240 catalog_manager.cc:5484] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 4 to 5, NON_VOTER e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) added. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:32.499962 17646 raft_consensus.cc:1203] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:18:32.569495 19385 ts_tablet_manager.cc:902] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: Initiating tablet copy from peer d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441)
I20240501 20:18:32.570674 19385 tablet_copy_client.cc:308] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.194:37441
I20240501 20:18:32.571358 17536 tablet_copy_service.cc:140] P d66e5484e3dd4fcbaa55ef26dd16ee35: Received BeginTabletCopySession request for tablet f492ab8a54f24904ac46820a0cef5e2f from peer e03cb8ab898b48938fe49f167a721d54 ({username='slave'} at 127.14.106.196:56767)
I20240501 20:18:32.571533 17536 tablet_copy_service.cc:161] P d66e5484e3dd4fcbaa55ef26dd16ee35: Beginning new tablet copy session on tablet f492ab8a54f24904ac46820a0cef5e2f from peer e03cb8ab898b48938fe49f167a721d54 at {username='slave'} at 127.14.106.196:56767: session id = e03cb8ab898b48938fe49f167a721d54-f492ab8a54f24904ac46820a0cef5e2f
I20240501 20:18:32.573522 17536 tablet_copy_source_session.cc:205] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:32.574689 19385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f492ab8a54f24904ac46820a0cef5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:32.578800 19385 tablet_copy_client.cc:783] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:32.579043 19385 tablet_copy_client.cc:647] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:32.580510 19385 tablet_copy_client.cc:523] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:32.582739 19385 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: Bootstrap starting.
I20240501 20:18:32.621290 19385 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:32.621682 19385 tablet_bootstrap.cc:492] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: Bootstrap complete.
I20240501 20:18:32.621912 19385 ts_tablet_manager.cc:1367] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: Time spent bootstrapping tablet: real 0.039s user 0.033s sys 0.005s
I20240501 20:18:32.622547 19385 raft_consensus.cc:348] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:32.622789 19385 raft_consensus.cc:729] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03cb8ab898b48938fe49f167a721d54, State: Initialized, Role: LEARNER
I20240501 20:18:32.622957 19385 consensus_queue.cc:260] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: NON_VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: true } }
I20240501 20:18:32.623446 19385 ts_tablet_manager.cc:1398] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 20:18:32.624289 17536 tablet_copy_service.cc:342] P d66e5484e3dd4fcbaa55ef26dd16ee35: Request end of tablet copy session e03cb8ab898b48938fe49f167a721d54-f492ab8a54f24904ac46820a0cef5e2f received from {username='slave'} at 127.14.106.196:56767
I20240501 20:18:32.624465 17536 tablet_copy_service.cc:433] P d66e5484e3dd4fcbaa55ef26dd16ee35: ending tablet copy session e03cb8ab898b48938fe49f167a721d54-f492ab8a54f24904ac46820a0cef5e2f on tablet f492ab8a54f24904ac46820a0cef5e2f with peer e03cb8ab898b48938fe49f167a721d54
I20240501 20:18:32.674898 19406 raft_consensus.cc:1050] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:32.675573 19406 consensus_queue.cc:237] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:32.678009 18171 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:32.678037 17383 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:32.678438 17515 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:32.678647 19408 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:32.679059 19319 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:32.679355 19406 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:32.690126 19319 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:32.691017 17384 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:32.691009 17515 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:32.694785 17236 catalog_manager.cc:5484] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:32.694706 18171 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:32.701483 17646 consensus_queue.cc:237] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:32.704206 17384 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:32.704550 18171 raft_consensus.cc:1261] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:32.704737 19408 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:32.705085 19406 consensus_queue.cc:1035] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:32.707201 19408 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:32.707818 18171 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:32.707871 17384 raft_consensus.cc:2932] T 16eb029403ad486f8f6da6b0d524baf0 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:32.710672 17229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16eb029403ad486f8f6da6b0d524baf0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:32.711223 17237 catalog_manager.cc:5484] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 6 to 7, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:32.715056 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet 16eb029403ad486f8f6da6b0d524baf0 with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:32.715869 19415 tablet_replica.cc:330] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:32.716213 19415 raft_consensus.cc:2229] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:32.716735 19415 raft_consensus.cc:2258] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:32.717828 19415 ts_tablet_manager.cc:1875] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:32.722039 19415 ts_tablet_manager.cc:1888] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:18:32.722146 19415 log.cc:1196] T 16eb029403ad486f8f6da6b0d524baf0 P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/16eb029403ad486f8f6da6b0d524baf0
I20240501 20:18:32.722920 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet 16eb029403ad486f8f6da6b0d524baf0 (table rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]) successfully deleted
I20240501 20:18:32.879248 19384 raft_consensus.cc:1050] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4: attempting to promote NON_VOTER be80849c49de4f188551b820fa459fd7 to VOTER
I20240501 20:18:32.879981 19384 consensus_queue.cc:237] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } }
I20240501 20:18:32.882529 17646 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 LEARNER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:32.882601 18171 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:32.882755 17515 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:32.883200 19383 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:32.883531 19268 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:32.883831 19384 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:32.887895 19268 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.889166 17515 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.892295 18171 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.894358 17238 catalog_manager.cc:5484] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.896024 17646 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.899314 17384 consensus_queue.cc:237] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } }
I20240501 20:18:32.901439 17646 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:18:32.901559 18171 raft_consensus.cc:1261] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:18:32.902091 19384 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:32.902521 19383 consensus_queue.cc:1035] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:32.904793 19384 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.906126 17646 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.908394 17226 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d941cb6f2beb456d96fa262f0c86e868 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:32.908547 18171 raft_consensus.cc:2932] T d941cb6f2beb456d96fa262f0c86e868 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.911496 17240 catalog_manager.cc:5484] T d941cb6f2beb456d96fa262f0c86e868 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:32.914938 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet d941cb6f2beb456d96fa262f0c86e868 with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:32.915333 19415 tablet_replica.cc:330] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:32.915526 19415 raft_consensus.cc:2229] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:32.915951 19415 raft_consensus.cc:2258] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:32.916864 19415 ts_tablet_manager.cc:1875] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:32.921222 19415 ts_tablet_manager.cc:1888] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:18:32.921334 19415 log.cc:1196] T d941cb6f2beb456d96fa262f0c86e868 P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/d941cb6f2beb456d96fa262f0c86e868
I20240501 20:18:32.921988 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet d941cb6f2beb456d96fa262f0c86e868 (table rebalance_test_table_2 [id=1615432bec3f4a83bf46183415fc1a59]) successfully deleted
I20240501 20:18:33.058970 17646 consensus_queue.cc:237] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:33.060813 18171 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:33.060851 17515 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:33.061363 19408 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:33.061642 19406 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:33.064334 19319 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.065003 17515 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.064991 18171 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.070065 17237 catalog_manager.cc:5484] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:33.075809 17646 consensus_queue.cc:237] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
I20240501 20:18:33.077567 18171 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:33.077812 17515 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:18:33.078161 17580 consensus_peers.cc:485] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 -> Peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): Couldn't send request to peer adc832fb6466498895e6f318e1c0ccb4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:33.078344 19408 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:33.078671 19406 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:33.081271 19408 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:33.083443 18171 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:33.083966 17229 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 49cf9d148ca64bdaa55fefb553dcb803 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:33.085621 17237 catalog_manager.cc:5484] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:33.087090 17515 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } } }
I20240501 20:18:33.113366 18172 raft_consensus.cc:1203] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:18:33.320533 17384 consensus_queue.cc:237] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:33.322636 17515 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:33.322664 18172 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:18:33.323263 19435 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:33.323609 19383 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:33.326323 19383 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.329576 17515 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.331190 17237 catalog_manager.cc:5484] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:33.338471 17384 consensus_queue.cc:237] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:33.343107 17515 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:18:33.343544 17320 consensus_peers.cc:485] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 -> Peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767): Couldn't send request to peer be80849c49de4f188551b820fa459fd7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:33.343855 19383 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:33.344014 18172 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:18:33.344559 19383 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:33.345664 18172 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.346930 19384 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:33.351490 18172 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:33.351994 17514 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:33.355336 17237 catalog_manager.cc:5484] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 7 to 8, NON_VOTER be80849c49de4f188551b820fa459fd7 (127.14.106.195) added. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } }
I20240501 20:18:33.358466 17226 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 50117416c3494c9c85c504ccd7df747c with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:18:33.496322 19484 ts_tablet_manager.cc:902] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Initiating tablet copy from peer be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767)
I20240501 20:18:33.497853 19484 tablet_copy_client.cc:308] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.195:43767
I20240501 20:18:33.498557 17667 tablet_copy_service.cc:140] P be80849c49de4f188551b820fa459fd7: Received BeginTabletCopySession request for tablet 49cf9d148ca64bdaa55fefb553dcb803 from peer adc832fb6466498895e6f318e1c0ccb4 ({username='slave'} at 127.14.106.193:58999)
I20240501 20:18:33.498733 17667 tablet_copy_service.cc:161] P be80849c49de4f188551b820fa459fd7: Beginning new tablet copy session on tablet 49cf9d148ca64bdaa55fefb553dcb803 from peer adc832fb6466498895e6f318e1c0ccb4 at {username='slave'} at 127.14.106.193:58999: session id = adc832fb6466498895e6f318e1c0ccb4-49cf9d148ca64bdaa55fefb553dcb803
I20240501 20:18:33.500638 17667 tablet_copy_source_session.cc:205] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:33.501643 19484 ts_tablet_manager.cc:1875] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:33.506119 19484 ts_tablet_manager.cc:1888] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:33.506340 19484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cf9d148ca64bdaa55fefb553dcb803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:33.508452 19484 tablet_copy_client.cc:783] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:33.508697 19484 tablet_copy_client.cc:647] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:33.510526 19484 tablet_copy_client.cc:523] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:33.512733 19484 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap starting.
I20240501 20:18:33.542838 19484 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:33.543190 19484 tablet_bootstrap.cc:492] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Bootstrap complete.
I20240501 20:18:33.543411 19484 ts_tablet_manager.cc:1367] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.004s
I20240501 20:18:33.543972 19484 raft_consensus.cc:348] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
I20240501 20:18:33.544184 19484 raft_consensus.cc:729] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Initialized, Role: LEARNER
I20240501 20:18:33.544369 19484 consensus_queue.cc:260] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: NON_VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: true } }
I20240501 20:18:33.544855 19484 ts_tablet_manager.cc:1398] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:18:33.545706 17667 tablet_copy_service.cc:342] P be80849c49de4f188551b820fa459fd7: Request end of tablet copy session adc832fb6466498895e6f318e1c0ccb4-49cf9d148ca64bdaa55fefb553dcb803 received from {username='slave'} at 127.14.106.193:58999
I20240501 20:18:33.545847 17667 tablet_copy_service.cc:433] P be80849c49de4f188551b820fa459fd7: ending tablet copy session adc832fb6466498895e6f318e1c0ccb4-49cf9d148ca64bdaa55fefb553dcb803 on tablet 49cf9d148ca64bdaa55fefb553dcb803 with peer adc832fb6466498895e6f318e1c0ccb4
I20240501 20:18:33.568221 19460 raft_consensus.cc:1050] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: attempting to promote NON_VOTER e03cb8ab898b48938fe49f167a721d54 to VOTER
I20240501 20:18:33.568868 19460 consensus_queue.cc:237] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:33.570983 17646 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:33.571066 17384 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:33.571090 18172 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 1 LEARNER]: Refusing update from remote peer d66e5484e3dd4fcbaa55ef26dd16ee35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:18:33.571645 19460 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:33.571993 19491 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:33.572305 19459 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:18:33.575253 19460 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.575947 17646 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.576433 17384 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.579239 18172 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:33.579914 17240 catalog_manager.cc:5484] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 5 to 6, e03cb8ab898b48938fe49f167a721d54 (127.14.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:33.618070 17515 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f"
dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39352
I20240501 20:18:33.618263 17515 raft_consensus.cc:595] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:33.721927 19516 ts_tablet_manager.cc:902] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Initiating tablet copy from peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049)
I20240501 20:18:33.723843 19516 tablet_copy_client.cc:308] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.106.193:35049
I20240501 20:18:33.724468 17404 tablet_copy_service.cc:140] P adc832fb6466498895e6f318e1c0ccb4: Received BeginTabletCopySession request for tablet 50117416c3494c9c85c504ccd7df747c from peer be80849c49de4f188551b820fa459fd7 ({username='slave'} at 127.14.106.195:53919)
I20240501 20:18:33.724644 17404 tablet_copy_service.cc:161] P adc832fb6466498895e6f318e1c0ccb4: Beginning new tablet copy session on tablet 50117416c3494c9c85c504ccd7df747c from peer be80849c49de4f188551b820fa459fd7 at {username='slave'} at 127.14.106.195:53919: session id = be80849c49de4f188551b820fa459fd7-50117416c3494c9c85c504ccd7df747c
I20240501 20:18:33.726511 17404 tablet_copy_source_session.cc:205] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:18:33.727488 19516 ts_tablet_manager.cc:1875] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:18:33.732069 19516 ts_tablet_manager.cc:1888] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:18:33.732323 19516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50117416c3494c9c85c504ccd7df747c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:18:33.734416 19516 tablet_copy_client.cc:783] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: tablet copy: Starting download of 0 data blocks...
I20240501 20:18:33.734674 19516 tablet_copy_client.cc:647] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:18:33.736294 19516 tablet_copy_client.cc:523] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:18:33.738586 19516 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Bootstrap starting.
I20240501 20:18:33.771085 19516 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:18:33.771477 19516 tablet_bootstrap.cc:492] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Bootstrap complete.
I20240501 20:18:33.771755 19516 ts_tablet_manager.cc:1367] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.004s
I20240501 20:18:33.772469 19516 raft_consensus.cc:348] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:33.772691 19516 raft_consensus.cc:729] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be80849c49de4f188551b820fa459fd7, State: Initialized, Role: LEARNER
I20240501 20:18:33.772876 19516 consensus_queue.cc:260] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } }
I20240501 20:18:33.773375 19516 ts_tablet_manager.cc:1398] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:18:33.774278 17404 tablet_copy_service.cc:342] P adc832fb6466498895e6f318e1c0ccb4: Request end of tablet copy session be80849c49de4f188551b820fa459fd7-50117416c3494c9c85c504ccd7df747c received from {username='slave'} at 127.14.106.195:53919
I20240501 20:18:33.774391 17404 tablet_copy_service.cc:433] P adc832fb6466498895e6f318e1c0ccb4: ending tablet copy session be80849c49de4f188551b820fa459fd7-50117416c3494c9c85c504ccd7df747c on tablet 50117416c3494c9c85c504ccd7df747c with peer be80849c49de4f188551b820fa459fd7
I20240501 20:18:33.853758 17515 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f"
dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46500
I20240501 20:18:33.853984 17515 raft_consensus.cc:595] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:33.897342 17384 raft_consensus.cc:1203] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:18:34.092447 17515 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f"
dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46522
I20240501 20:18:34.092671 17515 raft_consensus.cc:595] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:18:34.131596 19564 raft_consensus.cc:979] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: : Instructing follower adc832fb6466498895e6f318e1c0ccb4 to start an election
I20240501 20:18:34.131759 19564 raft_consensus.cc:1067] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Signalling peer adc832fb6466498895e6f318e1c0ccb4 to start an election
I20240501 20:18:34.132457 17384 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f"
dest_uuid: "adc832fb6466498895e6f318e1c0ccb4"
from {username='slave'} at 127.14.106.194:36253
I20240501 20:18:34.132689 17384 raft_consensus.cc:482] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:18:34.132803 17384 raft_consensus.cc:3037] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:34.134850 17384 raft_consensus.cc:504] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:34.135645 17384 leader_election.cc:288] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 2 election: Requested vote from peers d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441), be80849c49de4f188551b820fa459fd7 (127.14.106.195:43767), e03cb8ab898b48938fe49f167a721d54 (127.14.106.196:34303)
I20240501 20:18:34.136010 17515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35"
I20240501 20:18:34.136124 17646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "be80849c49de4f188551b820fa459fd7"
I20240501 20:18:34.136188 18172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f492ab8a54f24904ac46820a0cef5e2f" candidate_uuid: "adc832fb6466498895e6f318e1c0ccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e03cb8ab898b48938fe49f167a721d54"
I20240501 20:18:34.136272 17515 raft_consensus.cc:3032] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:18:34.136361 17646 raft_consensus.cc:3037] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:34.136394 17515 raft_consensus.cc:729] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d66e5484e3dd4fcbaa55ef26dd16ee35, State: Running, Role: LEADER
I20240501 20:18:34.136405 18172 raft_consensus.cc:3037] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:34.136727 17515 consensus_queue.cc:260] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:34.137174 17515 raft_consensus.cc:3037] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:18:34.138396 17646 raft_consensus.cc:2454] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 2.
I20240501 20:18:34.139223 18172 raft_consensus.cc:2454] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 2.
I20240501 20:18:34.139758 17317 leader_election.cc:302] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: adc832fb6466498895e6f318e1c0ccb4, be80849c49de4f188551b820fa459fd7, e03cb8ab898b48938fe49f167a721d54; no voters:
I20240501 20:18:34.140059 17515 raft_consensus.cc:2454] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc832fb6466498895e6f318e1c0ccb4 in term 2.
I20240501 20:18:34.140138 19383 raft_consensus.cc:2781] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:18:34.140515 19383 raft_consensus.cc:686] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 2 LEADER]: Becoming Leader. State: Replica: adc832fb6466498895e6f318e1c0ccb4, State: Running, Role: LEADER
I20240501 20:18:34.140838 19383 consensus_queue.cc:237] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:34.143404 17237 catalog_manager.cc:5484] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: term changed from 1 to 2, leader changed from d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) to adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193). New cstate: current_term: 2 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:18:34.167973 17646 raft_consensus.cc:1203] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:18:34.394246 19319 raft_consensus.cc:1050] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7: attempting to promote NON_VOTER adc832fb6466498895e6f318e1c0ccb4 to VOTER
I20240501 20:18:34.394979 19319 consensus_queue.cc:237] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:34.397286 17384 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEARNER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:34.397356 17515 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:34.397370 18172 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:34.397890 19319 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:34.398247 19586 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:34.398593 19542 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:34.407836 19542 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:34.409022 17515 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:34.409175 18172 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:34.413488 17238 catalog_manager.cc:5484] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 reported cstate change: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:34.414600 17384 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:34.418606 17646 consensus_queue.cc:237] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } }
I20240501 20:18:34.420663 17384 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:18:34.420814 18172 raft_consensus.cc:1261] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer be80849c49de4f188551b820fa459fd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:18:34.421277 19586 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:34.421660 19542 consensus_queue.cc:1035] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:34.423872 19515 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:34.426761 17229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49cf9d148ca64bdaa55fefb553dcb803 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:34.427433 17384 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P adc832fb6466498895e6f318e1c0ccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:34.427388 18172 raft_consensus.cc:2932] T 49cf9d148ca64bdaa55fefb553dcb803 P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } } }
I20240501 20:18:34.428084 17236 catalog_manager.cc:5484] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 reported cstate change: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "be80849c49de4f188551b820fa459fd7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:34.431831 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cf9d148ca64bdaa55fefb553dcb803 with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:34.433430 19590 tablet_replica.cc:330] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:34.433710 19590 raft_consensus.cc:2229] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:34.434237 19590 raft_consensus.cc:2258] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:34.435047 19590 ts_tablet_manager.cc:1875] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:34.439245 19590 ts_tablet_manager.cc:1888] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:18:34.439381 19590 log.cc:1196] T 49cf9d148ca64bdaa55fefb553dcb803 P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/49cf9d148ca64bdaa55fefb553dcb803
I20240501 20:18:34.440158 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet 49cf9d148ca64bdaa55fefb553dcb803 (table rebalance_test_table_1 [id=5fdae3ea335840a0aaec4354d538eb3d]) successfully deleted
I20240501 20:18:34.531587 18172 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:18:34.532222 19569 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:34.604234 17646 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:18:34.604887 19569 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:34.609249 17515 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:18:34.609809 19383 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:18:34.613492 17384 consensus_queue.cc:237] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } }
I20240501 20:18:34.615252 17646 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:18:34.615293 18172 raft_consensus.cc:1261] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:18:34.615772 19569 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:34.616053 19383 consensus_queue.cc:1035] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:18:34.618170 19569 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:34.618827 18172 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P e03cb8ab898b48938fe49f167a721d54 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:34.619254 17646 raft_consensus.cc:2932] T f492ab8a54f24904ac46820a0cef5e2f P be80849c49de4f188551b820fa459fd7 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } }
I20240501 20:18:34.622242 17226 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f492ab8a54f24904ac46820a0cef5e2f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:34.623323 17236 catalog_manager.cc:5484] T f492ab8a54f24904ac46820a0cef5e2f P adc832fb6466498895e6f318e1c0ccb4 reported cstate change: config changed from index 6 to 8, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 2 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:18:34.627810 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet f492ab8a54f24904ac46820a0cef5e2f with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:34.628186 19590 tablet_replica.cc:330] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:34.628360 19590 raft_consensus.cc:2229] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:34.628549 19590 raft_consensus.cc:2258] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:34.629487 19590 ts_tablet_manager.cc:1875] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:34.633817 19590 ts_tablet_manager.cc:1888] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 20:18:34.633939 19590 log.cc:1196] T f492ab8a54f24904ac46820a0cef5e2f P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/f492ab8a54f24904ac46820a0cef5e2f
I20240501 20:18:34.634555 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet f492ab8a54f24904ac46820a0cef5e2f (table rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]) successfully deleted
I20240501 20:18:34.778834 19565 raft_consensus.cc:1050] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4: attempting to promote NON_VOTER be80849c49de4f188551b820fa459fd7 to VOTER
I20240501 20:18:34.779502 19565 consensus_queue.cc:237] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } }
I20240501 20:18:34.781621 17646 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 LEARNER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:34.781963 17515 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:34.782039 18172 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:18:34.782202 19623 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:34.782692 19565 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:34.783028 19383 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:18:34.785759 19383 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:34.786289 18172 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:34.790688 17238 catalog_manager.cc:5484] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:34.792919 17646 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, be80849c49de4f188551b820fa459fd7 (127.14.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:34.795184 17384 consensus_queue.cc:237] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } }
I20240501 20:18:34.797677 18172 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:18:34.798278 19623 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:34.798422 17645 raft_consensus.cc:1261] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Refusing update from remote peer adc832fb6466498895e6f318e1c0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:18:34.799371 19383 consensus_queue.cc:1035] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:18:34.801162 19623 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P adc832fb6466498895e6f318e1c0ccb4 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:34.802048 18172 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:34.802282 17645 raft_consensus.cc:2932] T 50117416c3494c9c85c504ccd7df747c P be80849c49de4f188551b820fa459fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:34.806267 17237 catalog_manager.cc:5484] T 50117416c3494c9c85c504ccd7df747c P e03cb8ab898b48938fe49f167a721d54 reported cstate change: config changed from index 9 to 10, VOTER d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194) evicted. New cstate: current_term: 1 leader_uuid: "adc832fb6466498895e6f318e1c0ccb4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } }
I20240501 20:18:34.808156 17226 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50117416c3494c9c85c504ccd7df747c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:18:34.810810 17494 tablet_service.cc:1514] Processing DeleteTablet for tablet 50117416c3494c9c85c504ccd7df747c with delete_type TABLET_DATA_TOMBSTONED (TS d66e5484e3dd4fcbaa55ef26dd16ee35 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49846
I20240501 20:18:34.811125 19590 tablet_replica.cc:330] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: stopping tablet replica
I20240501 20:18:34.811301 19590 raft_consensus.cc:2229] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:18:34.811471 19590 pending_rounds.cc:62] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: Trying to abort 1 pending ops.
I20240501 20:18:34.811581 19590 pending_rounds.cc:68] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7022979951735123968 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "50117416c3494c9c85c504ccd7df747c" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: NON_VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "adc832fb6466498895e6f318e1c0ccb4" member_type: VOTER last_known_addr { host: "127.14.106.193" port: 35049 } } peers { permanent_uuid: "d66e5484e3dd4fcbaa55ef26dd16ee35" member_type: VOTER last_known_addr { host: "127.14.106.194" port: 37441 } attrs { replace: true } } peers { permanent_uuid: "e03cb8ab898b48938fe49f167a721d54" member_type: VOTER last_known_addr { host: "127.14.106.196" port: 34303 } attrs { promote: false } } peers { permanent_uuid: "be80849c49de4f188551b820fa459fd7" member_type: VOTER last_known_addr { host: "127.14.106.195" port: 43767 } attrs { promote: false } } } }
I20240501 20:18:34.811902 19590 raft_consensus.cc:2894] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240501 20:18:34.812019 19590 raft_consensus.cc:2866] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:18:34.812157 19590 raft_consensus.cc:2258] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:18:34.812925 19590 ts_tablet_manager.cc:1875] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:18:34.817180 19590 ts_tablet_manager.cc:1888] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:18:34.817289 19590 log.cc:1196] T 50117416c3494c9c85c504ccd7df747c P d66e5484e3dd4fcbaa55ef26dd16ee35: Deleting WAL directory at /tmp/dist-test-task8oH_XT/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714594103620019-14763-0/raft_consensus-itest-cluster/ts-1/wal/wals/50117416c3494c9c85c504ccd7df747c
I20240501 20:18:34.817971 17227 catalog_manager.cc:4830] TS d66e5484e3dd4fcbaa55ef26dd16ee35 (127.14.106.194:37441): tablet 50117416c3494c9c85c504ccd7df747c (table rebalance_test_table_0 [id=1146a6e6c6f74cf7b467faaf86e04db3]) successfully deleted
I20240501 20:18:35.274727 17363 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.303433 17626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.307009 17494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:35.319818 18152 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:35.466199 17363 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.476190 18152 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:35.482769 17626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.494081 17494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:35.631042 17363 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.643538 17626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.645998 18152 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:35.649698 17494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:35.803117 17363 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.812124 17626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.823944 17494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:35.824940 18152 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:35.958647 17363 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.969465 17626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:18:35.988828 17494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:18:35.992178 18152 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:18:36.179579 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 17302
I20240501 20:18:36.199144 17424 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:36.367666 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 17302
W20240501 20:18:36.396883 18107 connection.cc:535] server connection from 127.14.106.193:44935 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:18:36.397343 17583 connection.cc:535] server connection from 127.14.106.193:58999 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:18:36.397570 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 17434
I20240501 20:18:36.403539 17556 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:18:36.417903 17580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:35049: connect: Connection refused (error 111)
W20240501 20:18:36.419400 17580 consensus_peers.cc:485] T 49cf9d148ca64bdaa55fefb553dcb803 P be80849c49de4f188551b820fa459fd7 -> Peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): Couldn't send request to peer adc832fb6466498895e6f318e1c0ccb4. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:35049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:18:36.431517 17580 consensus_peers.cc:485] T 16eb029403ad486f8f6da6b0d524baf0 P be80849c49de4f188551b820fa459fd7 -> Peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): Couldn't send request to peer adc832fb6466498895e6f318e1c0ccb4. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:35049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:18:36.455888 17580 consensus_peers.cc:485] T b31922f4bb7042f4bfc2a7bb03589e3c P be80849c49de4f188551b820fa459fd7 -> Peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): Couldn't send request to peer adc832fb6466498895e6f318e1c0ccb4. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:35049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:18:36.480334 18105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.106.193:35049: connect: Connection refused (error 111)
W20240501 20:18:36.481817 18105 consensus_peers.cc:485] T 3fd75008254c4aef9740a5beb4f6288d P e03cb8ab898b48938fe49f167a721d54 -> Peer adc832fb6466498895e6f318e1c0ccb4 (127.14.106.193:35049): Couldn't send request to peer adc832fb6466498895e6f318e1c0ccb4. Status: Network error: Client connection negotiation failed: client connection to 127.14.106.193:35049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:18:36.568395 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 17434
I20240501 20:18:36.598044 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 17565
I20240501 20:18:36.603937 17686 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:36.771533 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 17565
I20240501 20:18:36.800534 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 18090
I20240501 20:18:36.806537 18211 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:36.956813 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 18090
W20240501 20:18:36.983456 17227 connection.cc:535] server connection from 127.14.106.196:42169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:18:36.983752 14763 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8oH_XT/build/asan/bin/kudu pid 17211
I20240501 20:18:36.984838 17272 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:18:37.150806 14763 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8oH_XT/build/asan/bin/kudu with pid 17211
2024-05-01T20:18:37Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1 (27325 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (48616 ms total)
[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (613567 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00>
1 FAILED TEST